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Is  DAWIA  Worth  It? 


An  Approach  to  Analyzing 
the  Impacts 

Dean  (Dusty)  Rhoads 


rhe  Defense  Acquisition  Workforce  Improvement  Act  (DAWIA)  has 
brought  change,  but  is  it  worth  it?  This  article  provides  information 
on  DA  WIA  and  suggests  an  approach  for  conducting  a  study  of 
the  impacts  of  implementing  DA  WIA. 


INTRODUCTION 

More  than  a  year  has  passed  since  the  last  mandatory  provision  of  the 
Defense  Acquisition  Workforce  Improvement  Act  (DAWIA)  became 
effective  (October  1993).  Results  are  beginning  to  surface,  although  the 
full  effects  of  DAWIA  implementation  will  not  be  known  until  the  rami¬ 
fications  of  a  more  highly  qualified  acquisition  workforce  have  worked 
their  way  through  the  system.  It  is  time  to  begin  asking  what  the  effects 
of  DAWIA  implementation  on  the  DoD  acquisition  system  and  workforce 
have  been  and  what  are  the  costs  associated  with  implementation.  As 
with  any  new  program  initiative,  structures  and  mechanisms  are  needed 
to  collect  the  necessary  data  and  to  identify  emerging  trends.  This 
article  identifies  an  approach  for  conducting  an  evaluation  of  DAWIA 
impacts  and  costs,  and  for  interpreting  the  results  based  on  proven  ana¬ 
lytical  techniques. 


Mr.  Rhoads  is  a  25  year  veteran  of  DoD  acquisition.  He  completed  a  suc¬ 
cessful  Air  Force  career  with  an  assignment  as  a  profes.sor  at  the  Defense 
Systems  Management  College  (DSMC)  where  he  initiated  and  taught  the 
Systems  Engineering  Management  Course.  Since  then  he  has  had  a  success¬ 
ful  career  in  industry’  and  is  now  at  ANSER,  a  public  service  research 
institute,  where  he  leads  a  Defense  Acquisition  Workforce  Improvement 
Act  project  for  the  Defense  Information  Systems  Agency.  Mr.  Rhoads  holds 
degrees  from  Iowa  State  University.  Boston  State  College,  and  The  George 
Washington  University.  He  is  also  a  graduate  of  the  Naval  War  College,  Air 
War  College,  DSMC  Program  Management  Course,  and  numerous  other 
DoD  acquisition  management  courses. 
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The  first  part  of  a  DAWIA  effects  study  would  be  a  performance 
evaluation;  that  is,  a  structured  assessment  of  the  Act’s  actual  or  poten¬ 
tial  impacts  on  the  acquisition  system,  its  processes,  people,  organiza¬ 
tions,  and  products.  The  primary  goal  would  be  to  assess  how  well  the 
objectives  of  DAWIA  are  being  realized.  The  evaluation  would  be  based 
on  identifying  criteria  for  success  by  establishing  suitable  measures  of 
effectiveness  (MOEs),  determining  which  MOEs  are  most  applicable  to 
the  problem  (i.e.,  have  the  highest  cause/effect  correlation),  and  differ¬ 
entiating  multiple  effects  from  multiple  causes.  The  second  part  of  the 
study  would  be  a  resource  analysis  keyed  to  the  measured  effect  of 
practical  constraints  (such  as  money,  other  resources,  and  time)  on  ex¬ 
pected  outcomes  and  achievable  capabilities.  In  combining  these  two 
parts,  this  study  resembles  several  other  types  of  analyses,  including 
tradeoff,  risk-return,  cost-benefit,  and  return-on-investment. 

STUDY  APPROACH 

A  carefully  selected  team  of  analysts  should  be  assembled  for  a  study  of 
this  scope.  It  should  possess  a  broad  mix  of  education,  training,  skills, 
and  experience  relevant  to  DAWIA,  defense  acquisition,  and  the  analy¬ 
sis  techniques  involved.  The  team  needs  to  build  synergy  and  carefully 
consider  all  aspects  of  a  problem  to  minimize  surprises  and  to  maintain 
objectivity.  At  all  steps  in  the  process,  close  and  frequent  contact  with 
the  DAWIA  stakeholders  should  be  maintained  to  ensure  that  the  analy¬ 
sis  remains  on  track  and  achieves  its  objectives. 

The  analysis  starts  with  the  problem  statement  as  the  premise  for  the 
study  and  follows  these  steps: 

1 .  Define  study  objective(s); 

2.  Define  problem  domain  and  boundaries; 

3.  Identify  MOEs; 

4.  Develop  model; 

5.  Identify  data  to  be  collected  and  sources  of  data; 

6.  Collect  data; 

7.  Analyze  and  interpret  data;  and 

8.  Report. 
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STUDY  OUTLINE 
Problem  Statement 

Since  full  DAWIA  implementation  was  mandated  to  occur  by  October 
1993,  a  detailed  analysis  of  its  impact  can  be  undertaken  now  to  develop 
the  methodology  and  models  and  to  collect  baseline  results.  Follow-up 
studies  can  then  be  conducted  annually  and  the  results  compared  to  the 
baseline  data  to  identify  trends  in  the  impacts  of  DAWIA  implementa¬ 
tion.  Annual  studies  can  be  accomplished  after  all  Services  and  agencies 
have  submitted  their  October  1994  DAWIA  reports  to  the  Defense  Man¬ 
power  Data  Center  (DMDC)  and  the  data  are  available  for  analysis. 

The  basic  process  for  the  initial  study  is  as  follows: 

•  Define  study  objective(s) 

The  purpose  of  analyzing  the  impacts  of  DAWIA  is  to  determine 
empirically  whether  its  objectives  are  being  achieved.  This  requires 
tracing  and  analyzing  the  Act’s  legislative,  statutory,  and  regulatory 
history  to  identify  the  underlying  expectations.  Study  questions  can 
then  be  formulated.  For  example,  what  is  the  effect  of  DAWIA 
implementation  on  the  DoD  acquisition  process?  What  is  the  re¬ 
turn  or  benefit  anticipated  from  implementing  DAWIA?  The  an¬ 
swers  to  these  questions  will  provide  decision  makers  with  perti¬ 
nent  information  to  support  informed  budgeting  decisions  for 
DAWIA. 

Performance  evaluation  in  this  case  would  be  accomplished  in 
two  phases,  implementation  and  effects.  For  the  implementation 
phase,  how  successfully  DAWIA  requirements  (e.g.,  the  require¬ 
ment  that  critical  acquisition  positions  be  filled  by  Defense  Acqui¬ 
sition  Corps  members)  have  been  implemented  across  all  DoD  ser¬ 
vices  and  agencies  would  be  evaluated.  In  the  effects  phase,  an 
attempt  would  be  made  to  quantify  the  impacts  of  DAWIA  imple¬ 
mentation  on  the  DoD  acquisition  process  (e.g.,  are  Defense  Ac¬ 
quisition  Corps  members  better  program  managers  than  pre- 
DAWIA  program  managers?). 

To  illustrate  the  methodology,  we  begin  from  the  premise  that 
an  objective  of  DAWIA  is  to  raise  the  qualifications  of  the  defense 
acquisition  workforce,  since  DAWIA  requires  acquisition  person¬ 
nel  to  have  more  education,  experience,  and  acquisition  training 
than  was  previously  required.  An  implied  assumption  is  that  a  more 
qualified  workforce  would  have  positive  impacts  on  DoD  acquisi¬ 
tion  programs  and  processes.  One  of  the  first  questions  to  answer  is 
this:  have  the  qualifications  of  the  defense  acquisition  workforce 
improved  since  DAWIA?  (In  statistical  analysis  terms,  this  is  an 
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“activity”  question.)  The  second,  more  difficult,  question  is  whether 
the  changes  in  the  defense  acquisition  workforce  have  had  any  im¬ 
pact  on  acquisition  programs  and/or  processes  (an  “outcome”  ques¬ 
tion).  This  second  analysis  can  only  be  concluded  after  determining 
that  there  have,  indeed,  been  changes  in  the  qualifications  of  the 
defense  acquisition  workforce  that  can  be  attributed  to  DAWIA 
implementation. 

A  resource  analysis  would  follow  each  of  the  performance  evalu¬ 
ation  phases  to  identify  the  costs  associated  with  bringing  about 
changes  in  the  qualifications  of  the  defense  acquisition  workforce, 
as  well  as  costs  saved  and/or  avoided  in  acquisition  programs  and 
processes  as  a  result  of  DAWIA. 

•  Deflne  problem  domain  and  boundaries 

The  problem’s  domain  and  boundaries  are  implicitly  defined  by  the 
objectives  of  the  analysis.  This  second  step  ensures  that  we  explic¬ 
itly  understand  what  is,  and  what  is  not,  part  of  the  problem.  Here 
we  would  determine  whether  to  answer  questions  such  as  these: 
what  is  the  nature  of  DAWIA’s  impact  on  the  DoD  acquisition 
process?  Has  it  been  effective?  Beneficial?  Worth  the  cost?  Bound¬ 
aries  must  be  identified  for  both  the  implementation  and  effects 
phases  of  the  performance  evaluation  and  resource  analysis. 

Objectives  must  be  structured  to  avoid  defining  problems  in  so 
broad  a  way  that  they  cannot  be  solved.  For  example,  one  broad 
objective  of  this  analysis  is  to  determine  if  DAWIA  has  had  posi¬ 
tive  effects  on  the  management  of  acquisition  programs.  To  be 
servicable,  this  objective  must  be  broken  down  into  multiple,  well- 
defined,  measurable  questions  that  can  be  answered  with  some  de¬ 
gree  of  certainty.  For  example,  docs  ACO  201  -  Intermediate  Sys¬ 
tems  Acquisition,  a  required  course  for  Level  II  certification  in  the 
career  fields  of  program  management  and  communications-comput- 
ers,  provide  effective  training  on  cost  control  measures?  If  the  ques¬ 
tions  are  not  appropriately  structured  and  bounded,  there  is  no  way  of 
assessing  whether  other  outside  influences  are  also  affecting  the  ob¬ 
served  results,  and  the  questions  become  impossible  to  answer. 

The  first  phase,  implementation,  would  be  easier  to  delimit  than 
the  less  well-defined  effects  phase.  Many  complex  variables  affect 
the  outcome  of  acquisition  programs,  some  of  which  are  beyond 
the  control  of  the  acquisition  workforce.  For  example,  if  an  acqui¬ 
sition  program  is  behind  schedule  and  over  cost,  is  it  due  to  prob¬ 
lems  with  its  acquisition  workforce  or  to  funding  perturbations  on 
Capitol  Hill  or  both?  These  kinds  of  considerations  would  require 
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time  to  sort  out  and  could  hinder  the  ability  to  assess  DAWIA 
effects  on  DoD  acquisition.  A  further  examination  of  this  analysis 
of  DAWIA  outcomes  may  reveal  that  finding  definitive  answers 
would  require  greater  investment  than  the  potential  benefits  war¬ 
rant.  It  may  be  more  beneficial  to  identify  a  series  of  indicators  of 
acquisition  program  success  rather  than  focus  efforts  on 
unachievable  results. 

•  Identify  measures  of  effectiveness  (MOEs) 

Which  workforce  qualifications  or  performance  objectives  need  to 
be  evaluated?  The  MOEs  would  be  different  for  each  phase  of  the 
study.  For  the  implementation  phase,  the  MOEs  would  be  mea¬ 
sures  of  workforce  performance  and  qualifications,  such  as  number 
of  critical  positions  identified,  critical  positions  filled  by  Corps-quali¬ 
fied  personnel,  and  requested  and/or  approved  waivers.  Education, 
experience,  and  acquisition  training  data  would  be  analyzed  to  iden¬ 
tify  trends  before  and  after  DAWIA. 

Identifying  MOEs  for  the  effects  phase  requires  more  study  and 
analysis  than  warranted  by  this  brief  outline.  A  key  question  in  the 
effects  phase  is  whether  a  post-DAWIA  workforce  is  accomplishing 
the  acquisition  business  of  DoD  more  effectively.  The  MOEs  in 
this  phase  would  be  much  more  difficult  to  collect  and  analyze. 
They  could  include  number  of  people  required  to  accomplish  vari¬ 
ous  acquisition  functions,  size  of  organizations,  and  length  and  com¬ 
plexity  of  acquisition  training  courses. 

From  a  resource  analysis  perspective,  in  the  implementation  phase 
the  study  would  measure  the  investment  cost,  and  in  the  effects 
phase,  the  return  on  investment.  The  MOEs  for  the  implementa¬ 
tion  phase  could  include  the  cost  of  acquisition  training,  the  cost  of 
reporting,  and  the  cost  of  maintaining  the  DAWIA  required  data. 
For  the  effects  phase,  MOEs  could  include  reduced  personnel  costs; 
avoidance  of  fraud,  waste,  and  abuse  costs;  and  cost  savings  through 
improved  performance.  All  MOEs  would  be  weighted  by  some  form  of 
dollar  and/or  time  factor  (i.e.,  before  and  after  DAWIA  comparisons 
of  schedules,  inspection  discrepancies,  resources  consumed,  etc.). 

•  Develop  model 

Models  of  the  implementation  phase  of  the  performance  evalua¬ 
tion  would  be  developed  to  assess  whether  or  not  DAWIA  has 
affected  the  “activity”  side  of  the  house  (i.e.,  changes  in  workforce 
qualifications).  Models  to  measure  the  effect  of  positive  activity 
results  on  acquisition  processes  and  programs  (outcomes)  would 
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also  be  developed.  The  focus  would  be  on  simple  models  that  illus¬ 
trate  major  trends  rather  than  on  complex  models,  which  tend  to 
lose  visible  results  in  too  much  detail.  All  models  would  be  ame¬ 
nable  to  sensitivity  analysis  and  “what  if’  exercises.  The  models 
would  also  identify  outside  factors  that  might  influence  outcomes. 
Examples  of  such  factors  include  acquisition  reform,  acquisition 
streamlining,  force  downsizing,  new  regulations,  and  budgetary  con¬ 
straints.  Technology  and  tools  could  also  be  mitigating  factors,  es¬ 
pecially  the  application  of  information  technologies  that  increase 
acquisition  process  efficiency  and  effectiveness. 

This  methodology  involves  identifying  dependent  and  indepen¬ 
dent  variables  and  their  relationships,  activities,  and  outcomes.  For 
example,  which  independent  variables  affect  the  qualifications  and 
performance  of  the  acquisition  workforce  (dependent  variables)? 
If  education,  acquisition  training,  and  experience  are  three  inde¬ 
pendent  variables,  what  is  the  relationship  between  them  and  the 
dependent  variables?  For  the  effects  phase  of  the  performance  evalu¬ 
ation,  the  dependent  variables  from  the  implementation  phase 
(workforce  qualifications)  would  become  the  independent  variables 
whose  effect  on  dependent  variables  (acquisition  processes  and  pro¬ 
grams)  would  be  identified.  Sensitivity  analyses  could  then  be  per¬ 
formed  by  varying  the  levels  of  workforce  qualifications  (i.e.,  mix  of 
education,  acquisition  training,  and  experience)  to  identify  the  ef¬ 
fects.  For  example,  from  a  return-on-investment  perspective,  what 
is  a  minimum  level  of  investment  (in  the  independent  variables)  to 
realize  any  effect,  or  what  level  of  investment  provides  the  greatest 
return,  or  what  level  of  investment  yields  the  greatest  percentage 
return? 

•  Identify  data  to  be  collected  and  sources  of  data 

The  MOEs,  the  activity  and  outcome  measures,  and  the  models 
would  be  the  determinants  in  deciding  what  data  to  collect  and 
analyze.  We  have  activity  measures  and  outcome  measures,  both  of 
which  have  MOEs  that  are  estimated  by  models.  In  the  case  of 
workforce  qualifications,  the  data  collected  would  include  educa¬ 
tional  degrees,  acquisition  course  completions,  and  experience.  No 
new  data  reporting  would  be  required.  Data  would  be  derived  from 
the  information  presently  collected  on  the  workforce  and  reported 
to  DMDC.  The  data  collected  would  be  both  pre-  and  post-DAWIA 
implementation  for  comparison  and  analysis.  The  DMDC  data  would 
be  compared  with  data  from  the  Defense  Acquisition  University 
(DAU)  to  identify  the  number  of  graduates  of  various  acquisition 
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training  courses  and  the  number  of  them  who  are  now  working  in 
acquisition  positions. 

•  Collect  data 

The  data  would  be  collected  and  used  to  validate  the  hypothesized 
models.  Data  already  being  reported  by  the  Services  and  agencies 
and  collected  by  DMDC  would  be  utilized  to  the  utmost;  no  new 
reporting  requirements  are  envisioned.  Most,  if  not  all,  data  needed 
to  evaluate  changes  in  workforce  qualifications  should  be  available 
from  existing  personnel  systems  and  can  be  collected  through  pro¬ 
grammed  queries  to  the  databases.  Cost  data  would  be  collected 
from  selected  financial  accounting  databases  in  DoD. 

•  Analyze  and  interpret  data 

Various  statistical  analysis  tools,  depending  on  the  models  chosen, 
would  be  employed.  By  using  the  tools  and  analyzing  the  results,  we 
would  identify  changes  in  workforce  qualifications  attributable  to 
DAWIA,  associate  causes  and  effects,  and  assess  the  value  of  ef¬ 
fects  in  relation  to  the  investment. 

•  Report 

Management  level  reports  summarizing  the  process  methodology 
and  interpreting  the  results  would  be  provided  at  the  conclusion  of 
the  analysis. 

SUMMARY 

The  DAWIA  implementation  is  driving  major  changes  in  the  way 
acquisition  careers  arc  managed  and  the  way  acquisition  professionals 
are  selected  for  assignments,  promotions,  and  advancement.  It  is  im¬ 
perative  that  DoD  decision  makers  fully  understand  the  impacts  of 
DAWIA  implementation,  not  only  on  acquisition  processes  and  pro¬ 
grams  but  also  on  the  people  involved.  A  performance  evaluation  and 
resource  analysis  would  help  DoD  ensure  that  DAWIA  implementation 
is  beneficial  to  both  its  people  and  its  processes  and  is  in  the  best  inter¬ 
ests  of  the  Government. 

This  article  outlines  a  methodology  to  provide  both  the  qualitative 
and  quantitative  feedback  that  DoD  executives  need  to  make  informed 
decisions  regarding  DAWIA.  The  study  outline  is  a  first  step  that  does 
not  answer  all  the  questions,  but  confronts  some  of  the  difficulties  in¬ 
volved  in  finding  answers.  The  effort  needs  to  be  undertaken  to  gener¬ 
ate  unbiased,  accurate,  in-depth,  and  pertinent  information  concerning 
the  merits  of  implementing  DAWIA  in  the  DoD. 
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rhe  ‘‘President’s  Blue  Ribbon  Commission  on  Defense  Management,  ” 
in  1986,  identified  three  military  programs,  the  Minuteman  I,  Polaris, 
and  the  Air  Launched  Cruise  Missile  (ALCM),  as  examples  of 
defense  programs  with  streamlined  procedures  that  achieved  accelerated 
(development)  schedules  typically  associated  with  successful  commercial 
programs.  The  Minuteman  I,  Polaris,  and  ALCM  acquisition  histories  were 
examined  to  identify  common  characteristics  and  to  determine  if  lessons 
learned  can  readily  be  transferred  to  other  DoD  programs  to  potentially 
improve  the  odds  of  program  success. 


INTRODUCTION 

In  1986,  the  “President’s  Blue  Ribbon  Commission  on  Defense  Man¬ 
agement,”  ,  stated  that,  “It  is  clear  that  major  savings  are  possible  in  the 
development  of  weapon  systems  if  the  Department  of  Defense  (DoD) 
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broadly  emulates  the  acquisition  procedures  used  in  outstanding  com¬ 
mercial  programs.” 

The  Commission  also  stated  that,  “It  is  clear  from  our  earlier  descrip¬ 
tion  that  defense  acquisition  typically  differs  from  this  commercial  model 
in  almost  every  respect.”  It  did  name,  however,  the  Minuteman  I,  Po¬ 
laris,  and  (post-Milestone  II)  long-range  Air  Launched  Cruise  Missile 
(ALCM)  as  examples  of  defense  programs  with  streamlined  procedures 
that  achieved  accelerated  (development)  schedules  typically  associated 
with  successful  commercial  programs. 

Only  the  post-Milestone  II  portion  of  the  ALCM  program  used  stream¬ 
lined  acquisition  procedures.  The  long-range  ALCM  program  (AGM- 
86B)  was  effectively  initiated  as  a  result  of  the  cruise  missile  Milestone 
II  decision  memorandum  on  January  14,  1977.  Prior  to  this  time,  the 
long-range  ALCM  had  only  been  a  paper  study.  Now  it  was  to  be  devel¬ 
oped  and  given  priority  over  the  existing  short-range  ALCM  (AGM- 
86A)  the  Air  Force  had  been  developing  (Conrow,  Smith,  and  Barbour, 
1982).  (Shortly  thereafter,  the  short-range  ALCM  program  ended.)  The 
ALCM  program  was  transferred  to  the  Joint  Cruise  Missiles  Project 
Office  (JCMPO)  as  a  result  of  the  Milestone  II  decision  memorandum. 
(The  JCMPO  was  also  created  as  a  result  of  the  Milestone  II  decision 
memorandum.)  However,  it  was  not  until  President  Jimmy  Carter  can¬ 
celed  the  B-IA  program  on  June  30,  1977  and  the  September  30,  1977 
memorandum  from  Undersecretary  of  Defense  for  Research  and  Engi¬ 
neering  (USDR&E)  Dr.  William  J.  Perry  to  the  Secretaries  of  the  Air 
Force  and  Navy,  that  the  JCMPO  was  given  sufficient  external  support 
in  order  to  implement  a  streamlined  acquisition  program  for  the  long- 
range  ALCM.  The  ALCM  flyoff  was  initiated  in  the  same  Dr.  Perry 
memorandum  eight  months  after  the  program  entered  Full-Scale  Devel¬ 
opment  (FSD,  now  Engineering  and  Manufacturing  Development  or 
EMD). 

The  development  program  acquisition  histories  of  the  Minuteman  I, 
Polaris  A-1,  and  ALCM  systems  were  examined  to  identify:  (1)  some 
common  characteristics  of  these  programs,  and  (2)  whether  or  not  the 
lessons  learned  can  be  transferred  to  other  DoD  programs  to  potentially 
improve  the  odds  of  program  success. 

COMPARISON  OF  MINUTEMAN  I,  POLARIS  A-1, 

AND  ALCM  DEVELOPMENT  PROGRAMS 

The  Minuteman  I  (Anderson,  1977),  Polaris  A-1  (Sapolslcy,  1972,  Navy 
Department,  1986  &  Mitchell,  1987)  and  ALCM  (Conrow,  Smith  & 
Barbour),  program  schedules  are  summarized  in  Figures  1,  2,  and  3, 
respectively. 
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Minuteman  I  Schedule 

Figura  1 
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Figure  1.  Minuteman  I  Schedule 


Polaris  Schedule 

Figure  2 
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Source:  H.  M.  Sapolsky,  "The  Polaris  System  Development."  Harvard  University  Press,  Cambridge,  1972; 

"FBM  Facts/Chronology:  Polaris,  Poseidon,  Trident,"  Strategic  Systems  Program  Office,  1986;  and 
CAPT  John  Mitchell.  Strategic  Systems  Program  Office,  private  communication,  January-March  1987. 


Figure  2.  Polaris  Schedule 
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ALCM  Development  History 

Figure  3 
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Figure  3.  ALCM  Development  History 


Priority  Rating 

Each  program  was  of  the  highest  national  importance  and  held  a  BRICK¬ 
BAT  DX  priority  rating.  This  rating  provided  assurance  for  availability 
of  materials,  components,  and  other  resources  (e.g.,  test  ranges)  in  the 
event  of  conflict  with  commercial  or  lesser  important  defense  contracts. 
The  Minuteman  I,  Polaris  (Fleet  Ballistic  Missile  Program),  and  ALCM 
were  assigned  this  rating  in  September  1959,  November  1955,  and  Feb¬ 
ruary  1978,  respectively. 

Program  Management  Autonomy 

Each  program  had  considerable  management  autonomy.  However,  this 
did  not  prevent  either  bureaucratic  encroachment  or  considerable  pro¬ 
grammatic  turbulence  from  occurring  during  missile  development  (Min¬ 
uteman  1)  or  following  development  completion  (Polaris  A-1  and  ALCM). 

Conflicts  between  programmatic  success  and  bureaucratic  success  ex¬ 
ist  and  generally  increase  as  a  program  nears  completion.  A  potential 
cause  of  this  problem  is  that  tendencies  tow'ards  suboptimization  inher- 
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ent  in  project  management  are  exacerbated  when  the  urgency  of  the 
mission  declines  (Sapolsky). 

In  the  Minuteman  I  case,  Lieutenant  General  Bernard  Schriever, 
USAF,  was  the  commander  of  the  Air  Force  Ballistic  Missile  Division, 
and  the  deputy  commander  of  the  Air  Research  and  Development  Cen¬ 
ter  (ARDC,  which  later  became  Air  Force  Systems  Command).  In 
essence,  Lt  Gen  Schriever  was  his  own  supervisor,  which  gave  him  very 
strong  control  over  funding  and  review  authority  through  1958  (early 
FSD  program  phase).  However,  upon  becoming  ARDC  commander  in 
March  1959,  Lt  Gen  Schriever  abolished  special  management  by  excep¬ 
tion  for  the  Minuteman  I  program  office  and  the  program  no  longer  pos¬ 
sessed  such  uncommon  reporting  and  protective  mechanisms  (Piper,  1962). 

Minuteman  I  management  independence  had  declined  considerably 
just  prior  to  the  March  1960  production  decision,  as  evidenced  in  an 
excerpt  of  a  letter  from  Lt  Gen  Schriever  to  General  Thomas  White 
(Chief  of  Staff,  USAF): 

...  To  insure  a  timely  and  effective  force,  our  weapon 
system  managers  must  possess  the  authority  to  effect  nec¬ 
essary  program  changes.  The  continual  encroachment 
upon  this  authority  is  of  great  concern  to  me,  and  I  urge 
that  you  consider  a  return  to  the  streamlined  manage¬ 
ment  principles  which  were  so  effective  in  the  Thor  and 
Atlas  programs.  (Piper) 

By  February  1961,  the  Minuteman  I  management  situation  had  be¬ 
come  so  difficult  that  Lt  Gen  Schriever  expressed  great  concern  to  Gen. 
White.  Lt  Gen  Schriever  felt  that  although  the  Minuteman  program  was 
a  high  risk  program  requiring  unusual  measures  in  all  respects  to  pro¬ 
tect  the  operational  dates,  the  program  was  being  managed  in  a  routine 
fashion  (Smith  1970). 

Similarly,  Major  General  Osmund  J.  Ritland,  Commander  of  the  Air 
Force  Ballistic  Missile  Division,  in  a  February  17,  1961  letter  to  Lt  Gen 
Schriever.  drew  attention  to  how  the  Navy  had  been  supporting  the 
Polaris  program.  Maj  Gen  Ritland  indicated  that  the  Navy  had  rallied  to 
support  Polaris  development,  and  that  there  had  been  no  change  in 
program  management  since  the  beginning  of  the  program.  The  program 
director  had  been  given  total  authority  and  resources  to  carry  out  his 
assignment.  For  this  “.  .  .  he  had  been  asked  for  only  one  thing — re¬ 
sults."  (Piper) 

The  most  important  Minuteman  1  management  strengthening  mea¬ 
sures  occurred  on  February  23,  1961.  General  White  informed  all  depu- 
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ties,  directors,  and  chiefs  of  comparable  offices  that  the  Minuteman 
undertaking  was  a  .  crash  program.”  Therefore,  all  matters  pertain¬ 
ing  Minuteman  high  level  program  review  were  given  “overriding  prior¬ 
ity,”  and  reviews  of  the  program  were  to  be  limited  to  a  single  review  in 
the  Air  Staff  with  ARDC  participation.  (Piper) 

In  the  Polaris  case,  the  Special  Projects  Office  quickly  learned  that  a 
reputation  for  managerial  efficiency  made  it  difficult  for  anyone  to  chal¬ 
lenge  the  Polaris  development  plans.  Using  the  Program  Evaluation  and 
Review  Technique  (PERT),  a  computerized  planning,  scheduling,  and 
control  technique  first  made  public  in  mid-1958,  the  Polaris  Manage¬ 
ment  Center,  and  other  management  innovations  produced  a  protective 
veneer  to  allow  Polaris  development  (Sapolsky).  The  degree  of  program 
protection  provided  by  these  innovations  was  of  equal  or  greater  value 
than  the  intrinsic  management  efficiency  benefits  of  the  techniques.  They 
allowed  the  technical  staff  to  work  relatively  unhindered  and  protected 
them  from  concerned  but  potentially  outside  officials. 

Decentralization  and  competition  were  key  components  of  the  Polaris 
management  strategy,  which  provided  nearly  self-regulating  control  over 
the  Polaris  development  and  its  developers.  Through  decentralization, 
authority  to  act  was  given  to  those  closest  to  the  problems,  yet  competi¬ 
tion  among  the  program  office  branches  and  contractors  assured  the 
central  staff  that  decisions  affecting  the  vital  needs  of  the  entire  system 
would  be  brought  to  their  attention  (Sapolsky).  However,  Department 
of  the  Navy  restructuring  in  1963  and  1966  eliminated  much  of  the 
Special  Project  Office  independence  and  management  by  exception 
(Sapolsky).  Although  this  did  not  greatly  affect  the  Polaris  series  of 
missiles  (A-1,  A-2,  and  the  initial  A-3),  it  did  impact  all  subsequent 
Navy  ballistic  missile  programs. 

An  Executive  Committee  (EXCOM)  was  established  by  USDR&E 
Dr.  Perr}^  in  September  1977  (early  in  FSD)  to  provide  programmatic 
and  fiscal  direction  to  monitor  the  progress  of  the  ALCM  flyoff  and 
other  cruise  missile  variants.  The  EXCOM  was  not  a  voting  group;  rather 
its  purpose  was  to  review  and  discuss  in  an  attempt  to  establish  a  con¬ 
sensus.  In  the  absence  of  a  consensus.  Dr.  Perry  would  act  as  required 
and  report  dissenting  opinions  to  the  Secretary  of  Defense  along  with 
recommendations  for  action.  Normal  channels  remained  open  to  the 
Serv'ices  to  express  dissent.  Another  EXCOM  feature  was  that  it  pro¬ 
vided  a  forum  for  an  expeditious  review  of  problem  areas.  In  addition, 
through  its  high-level  OSD  and  Service  membership  and  the  use  of 
action  item  assignments.  EXCOM  interaction  with  the  JCMPO  could 
potentially  minimize  program  cost  and  schedule  risk  (Conrow,  Smith  & 
Barbour). 
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The  EXCOM  was  a  key  element  in  the  JCMPO  management  approach. 
In  practice,  the  EXCOM  served  several  functions.  One  function  was  to 
provide  a  periodic  and  structured  forum  for  examining  problem  areas. 
Another  function  was  that  the  EXCOM  members  had  enough  authority 
to  resolve  problems  quickly,  including  resolution  of  funding  shortages 
(Conrow,  Smith,  &.  Barbour). 

The  ALCM  began  transitioning  back  to  the  Air  Force  from  the  JCMPO 
following  its  Milestone  III  review  in  March  1980.  The  cruise  missile 
project  EXCOM  was  discontinued  after  its  final  meeting  on  January  8, 
1981.  After  that,  the  JCMPO  director  had  no  effective  formal  mecha¬ 
nism  for  resolving  issues  between  the  Air  Force  and  Navy. 

Although  the  demise  of  the  EXCOM  did  not  affect  ALCM  develop¬ 
ment,  it  did  potentially  impact  development  of  the  Air  Force  Ground 
Launched  Cruise  Missile  and  the  Medium  Range  Air-to-Surface  Missile 
programs.  Another  effect  was  a  psychological  one  in  the  minds  of  some 
JCMPO  staff  members  and  associated  service  officials  and  some  per¬ 
sonnel  took  this  action  to  mean  in  part  that  the  level  of  Office  of  the 
Secretary  of  Defense  (OSD)  support  for  the  cruise  missile  project  had 
diminished.  Although  impossible  to  quantify,  this  factor  undoubtedly 
had  some  nonbencficial  impact  on  the  cruise  missile  project  (Conrow, 
Smith.  &  Barbour). 

Early  Program  Support 

None  of  the  three  programs  received  substantial  support  until  the  middle 
development  stages.  For  example,  the  Minuteman  I  and  Polaris  pro¬ 
grams  were  initiated  after  the  four  “approved”  ballistic  missile  pro¬ 
grams — Atlas  Intercontinental  Ballistic  Missile  (ICBM),  Titan  ICBM 
(backup),  Thor  Intermediate  Range  Ballistic  Missile  (IRBM),  and  Jupi¬ 
ter  IRBM — were  identified  by  President  Dwight  D.  Eisenhower  and  the 
DoD  in  September  1955  (Sapolsky).  In  addition,  early  attempts  to  accel¬ 
erate  the  Minuteman  program  schedule  were  not  well  received.  Just 
prior  to  the  Minuteman  I  FSD  start  and  11  months  after  the  first  Soviet 
Sputnik  was  launched,  W.  M.  Holaday,  Department  of  Defense  Director 
of  Guided  Missiles,  wrote  to  James  H.  Douglas.  Secretary  of  the  Air 
Force,  on  September  17,  1958,  noting  that  the  Minuteman  program; 

...  is  not  in  consonance  with  my  desire  for  an  orderly 
step-wise  development  program.  The  plan  submitted  is 
characterized  by  the  compressed  development  schedules 
associated  with  the  so-called  crash  programs  such  as  At¬ 
las  and  Titan  which,  while  justified  by  the  urgency  of  the 
requirement  for  an  early  ICBM  capability,  are  not  con- 
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ducive  to  maximizing  operational  effectiveness  or  mini¬ 
mizing  costs.  (Piper) 

Minuteman  I  had  a  less  hectic  development  program  than  the  Polaris 
A-1,  partly  because  it  was  in  the  awkward  position  of  being  competitive 
with  more  advanced  liquid-fuel  ballistic  missiles  that  were  being  devel¬ 
oped  by  the  Air  Force.  Essentially,  all  the  critical  uncertainties  of  Min¬ 
uteman  technology  were  reasonably  well  in  hand  by  1957,  but  activating 
a  full-scale  development  program  would  inevitably  cause  a  diversion  of 
effort  from  the  other  ballistic  missile  programs  to  which  the  Air  Force 
had  commitments.  Before  Sputnik  cut  the  purse  strings,  Minuteman  I 
could  have  been  developed  only  at  the  price  of  limiting  expenditures  of 
the  Atlas  or  Titan  programs  (Perry,  1967).  The  effect  of  the  Sputnik 
furor  in  late  1957  and  of  the  political  squabbling  that  sputtered  through 
the  next  three  years  was,  in  part,  the  accelerated  development  of  the 
Minuteman  I  and  Polaris  A-1  programs. 

Similarly,  by  March  1957  the  Navy’s  Special  Projects  Office  had  settled 
on  the  general  specifications  of  the  Polaris  missile,  submarine, 
undersurface  launch  system,  and  related  components.  Acceleration  of 
the  original  program  and  substantial  funding  authorizations  followed 
Sputnik.  Cutting  rather  more  than  two  years  from  the  earlier  schedules 
was  achieved  by  compressing  schedules,  eliminating  test  sequences,  and 
by  relaxing  both  the  range  and  accuracy  specifications  (Perry). 

The  long-range  ALCM  faced  initial  opposition  from  parts  of  the  Air 
Force  in  favor  of  its  much  higher  priority  B-IA  coupled  with  the  short- 
range  ALCM  prior  to  President  Carter’s  cancellation  of  the  B-:1A  on 
June  30,  1977  (Werrell,  1985). 

In  addition,  none  of  the  programs  had  considerable  opposition  from 
the  scientific  community  beyond  the  early  development  stage. 

Program  Funding  and  Funding  Turbulence 

While  each  program  received  the  necessary  funding,  potential  funding 
shortfalls  occurred,  producing  program  turbulence  in  the  short  run. 

In  the  Minuteman  I  case,  an  accelerated  development  plan  was  ap¬ 
proved  on  May  20,  1959  (mid-way  through  FSD).  By  November  1959, 
the  Minuteman  I  contractors  were  reporting  the  potential  for  large  cost 
increases.  The  cost  increases  were  the  result  of  externally  directed  changes 
to  the  Minuteman  program,  including  the  accelerated  schedule.  Initially, 
the  corresponding  increases  in  the  program  budget  did  not  match  these 
externally  directed  changes,  and  substantial  increases  in  Minuteman  I 
funding  were  necessary  during  the  early  portion  of  the  missile  produc¬ 
tion  phase  in  FY60-FY62  (Piper). 
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In  the  Polaris  case: 

An  unexpected  technical  crisis  would  force  internal  re¬ 
programming  of  funds  to  be  undertaken.  But  inevitably 
money  for  all  important  activities  was  found,  even  if  the 
lost  time  could  not  always  be  recovered.  Program  offi¬ 
cials  learned  quickly  to  request  generous  contingency 
appropriations.  (Sapolsky) 

On  December  9,  1957  (at  the  start  of  FSD),  the  Secretary  of  Defense 
authorized  acceleration  of  the  Polaris  program  to  deploy  the  first  Po¬ 
laris  weapon  system  in  1960.  Shortly  thereafter,  on  February  12,  1958, 
President  Eisenhower  signed  the  FY58  Supplemental  Appropriation  Act, 
including  funds  for  construction  of  the  first  three  Polaris  submarines. 
Construction  had  begun  in  Januaiy'  1958  using  funds  “borrowed”  from 
other  Na\7  programs  (Navy  Department,  1990). 

In  the  ALCM  case,  the  FSD  program  flyoff  completion  date  slipped 
approximately  three  months  during  the  course  of  the  competition  (from 
November  1979  to  February  1980)  because  of  the  late  receipt  of  the 
FY78  supplemental  appropriation  (Conrow,  Smith,  &  Barbour),  which 
was  dedicated  to  the  ALCM  program. 

While  it  is  difficult  to  estimate  accurately  the  impact  of  program  ac¬ 
celeration  on  the  development  cost,  the  resulting  increases  were  not 
minor  for  Minuteman  I  and  ALCM.  In  the  Minuteman  I  case,  it  is 
estimated  that  program  acceleration  added  roughly  45%  to  the  develop¬ 
ment  phase  cost,  and  was  used  for  overtime  for  contractor  employees 
and  dual  sources  for  critical  items  and  tests  (Anderson).  In  the  ALCM 
case,  a  41%  cost  growth  occurred  in  the  development  phase — the  largest 
contributors  to  this  growth  were  the  accelerated  FSD  schedule  and  some 
additional  requirements  imposed  during  the  flyoff  (Conrow,  Smith,  & 
Barbour). 

Necessary  Technology  Advancement 

Each  program  represented  a  moderate  technology  advance  required 
across  the  entire  system,  with  a  considerable  advance  in  the  state  of  the 
art  required  for  only  a  few  sub.systcms.  In  addition,  each  program  was 
the  beneficiary  of  important  advances  made  from  prior  technology,  de¬ 
velopment.  or  operational  programs. 

For  example,  in  the  Minuteman  1  and  Polaris  A-1  programs,  impor¬ 
tant  advances  had  already  been  made  in  the  Atlas,  Thor,  and  Titan 
ballistic  missile  programs,  including  reentry  physics,  development  of  elec¬ 
tronic  controls,  and  inertial  guidance.  In  addition,  key  solid  propulsion 
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research  had  been  started  in  the  mid  1950s  that  provided  an  excellent 
bridge  to  Minuteman  I  and  Polaris  A-1  requirements.  The  “break¬ 
through”  for  each  came  in  1955  (prior  to  the  commencement  of  either 
program)  with  demonstrations  that  large-grain,  double-base  solid  pro¬ 
pellants  could  be  reliably  ignited  and  burned  (Perry) 

In  the  ALCM  case,  key  turbofan  engine  development  had  begun  al¬ 
most  five  years  before  the  January  1977  Milestone  II  decision  to  initiate 
development  of  the  long-range  ALCM  (Conrow,  Smith,  &  Barbour). 
Likewise,  the  guidance  system  Inertial  Navigation  Element  (INE)  was 
closely  related  to  a  unit  that  had  already  been  developed  and  tested  for 
the  F-15  and  other  military  aircraft.  In  effect,  the  technical  characteris¬ 
tics  of  both  the  ALCM  engine  and  INE  were  well  in  hand  prior  to  the 
cruise  missile  Milestone  II  decision  on  January  14,  1977. 

Common  areas  of  technology  advancement  used  by  all  three  pro¬ 
grams  included  improved  propulsion  and  warheads. 

Some  common  key  technology  areas  for  the  Minuteman  I  and  Polaris 
A-1  that  required  moderate  development  included:  (1)  high  energy,  solid 
propellants  with  advanced  binders  and  additives,  and  uniform  character, 
which  provided  improved  thrust  and  reliability;  (2)  improved  materials 
(e.g.,  with  increased  strength  and  reduced  weight),  which  contributed  to 
increased  range;  (3)  thrust  vector  control  and  thrust  termination  de¬ 
vices,  which  improved  missile  accuracy;  (4)  ablative  reentry  vehicle  de¬ 
velopment,  which  reduced  payload  weight  and  reentry  dispersion,  and 
increased  accuracy;  and  (5)  reduced  thermonuclear  warhead  weight, 
which  reduced  payload  weight,  thus  increasing  range,  for  a  given  war¬ 
head  yield. 

The  ALCM  key  technology  areas  included:  (1)  a  low  specific  fuel 
consumption,  high  thrust,  small  volume  turbofan  engine  with  high  alti¬ 
tude  startup,  which  increased  range;  (2)  a  special  warhead  design;  (3)  a 
low  observables  air  vehicle,  using  a  special  airframe,  missile  radar  altim¬ 
eter,  and  engine  design,  which  increased  survivability;  and  (4)  terrain 
contour  matching  and  terrain  following  software,  which  increased  mis¬ 
sile  accuracy  and  survivability. 

Missile  Thrust/Weight  Ratio 

The  initial  thrust/weight  ratio  a.ssociated  with  engine  propulsion,  coupled 
with  missile  weight,  proved  to  be  optimistic  for  each  program.  This  led 
to  a  decrease  in  range  for  the  first  wing  of  Minuteman  I  deployed  and 
the  Polaris  A-1,  and  is  indicative  of  an  initial  set  of  system  requirements 
that  exceeded  the  feasible  level  of  performance  that  could  be  achieved 
for  the  estimated  level  of  cost  and  schedule. 

Sacrifices  versus  desired  performance  levels  were  made  for  each  pro- 
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gram  in  order  to  meet  the  initial  deployment  schedule.  For  the  Minute- 
man  I  and  Polaris  A-1,  a  reduction  in  performance  occurred  along  with 
an  increase  in  development  phase  cost  in  the  Minuteman  I.  Approxi¬ 
mately  60  percent  of  the  range  reduction  in  the  first  Minuteman  I  wing 
was  corrected  in  the  second  Minuteman  I  wing  deployed  in  1963,  and 
virtually  all  of  the  Polaris  A-1  missile  range  reduction  was  removed  in 
the  Polaris  A-2  missile.  The  range  (performance)  reduction  was  accepted 
in  the  first  Minuteman  I  and  Polaris  A-1  missiles  deployed  in  order  to 
reduce  program  schedule  risk.  In  the  ALCM  case,  no  early  attempt  was 
made  to  correct  the  problem  since  the  demonstrated  performance  level 
was  adequate.  The  government  emphasis  in  the  ALCM  program  was  to 
ensure  the  eventual  production  deliveiw  schedule  while  maintaining  de¬ 
velopment  phase  cost  and  performance. 

Use  of  Parallel  R&D 

Parallel  research  and  development  (R&D)  and  simultaneous  explora¬ 
tion  of  several  alternatives  were  used  extensively  as  risk  reduction  mea¬ 
sures  for  selected  key  technology  areas  that  required  considerable  ad¬ 
vances  in  the  state  of  the  art.  Parallel  R&D  had  been  recognized  by  this 
time  as  a  key  tool  for  reducing  program  risk  when  technology  advances 
were  required,  particularly  on  a  short  schedule  (Klein,  Meckling,  & 
Mesthene,  1958). 

The  extended  parallel  R&D  present  in  the  Minuteman  I,  Polaris  A-1, 
and  ALCM  programs  greatly  reduced  the  resulting  risk  associated  with 
the  relatively  short  program  schedule  length,  coupled  with  the  technol¬ 
ogy  development  that  was  necessary. 

For  example,  the  Minuteman  I  program  had  as  many  as  three  differ¬ 
ent  contractors  developing  missile  propulsion  stages  during  the  period 
from  1956  to  1961,  as  shown  in  Figure  4  (Anderson).  The  parallel  R&D 
efforts  were  maintained  until  (and  in  the  case  of  the  third  stage,  after) 
the  program  production  decision  in  March  1960. 

For  Polaris: 

In  the  launch  area,  11  different  methods  of  ejecting  a 
missile  from  a  submerged  submarine  were  said  to  have 
been  simultaneously  considered.  Similarly,  in  the  naviga¬ 
tion  area  at  least  two  teams  approached  the  problem  of 
developing  an  inertial  navigation  system  and  several  sub¬ 
stitute  navigation  schemes  were  also  explored.  (Sapolsky) 

In  the  ALCM  program,  the  engine  and  INE  were  well  advanced  in 
development  prior  to  initiating  the  long-range  ALCM  program  in  Janu- 
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Minuteman  I  State-Of-The-Art 
Propulsion  Technology 

EXAMPLE  OF  EXTENSIVE  PARALLEL  R&D 


•  CONTRACTOR  EFFORT 

•  AEROJET  STG  I.  II,  III 


FEASIBILITY 

STG  II  MM  CONTRACT 

•  PHILLIPS/ASTROOVNE 


FEASIBILITY 
•  THIOKOL  STG  I.  II.  Ill 


FEASIBILITY 

STG  I  MM  CONTRACT 

•  HERCULES  STO  III 


FEASIBILITY 

STG  III  MM  CONTRACT 


Sourca:  Robarl  C.  Andanon.  'Tha  Acquisition  Procass.  Minuteman  l-MX,’  TRW,  Inc., 
AupusI  1977,  pg,  10. 


Figure  4.  Minuteman  I  State-of-the-Art  Propulsion  Technology 


ary  1977.  Each  item  had  undergone  parallel  development  with  two  con¬ 
tractors  culminating  with  a  selected  design  prior  to  beginning  FSD.  For 
example,  the  engine  and  INE  down  select  to  a  single  development  con¬ 
tractor  occurred  in  April  1973  and  October  1975,  respectively.  However, 
the  required  ALCM  deployment  schedule  necessitated  increasing  the 
production  rate  for  the  engine  and  INE.  An  identical  design  dual  source 
production  competition  strategy  was  established  during  FSD  for  the 
engine  and  INE  in  August  1978  and  October  1978,  respectively,  to  meet 
the  ALCM  deployment  schedule  (Conrow,  Smith,  &  Barbour). 

Development  Phase  Flight  Tests 

Although  the  Minuteman  I,  Polaris  A-1,  and  ALCM  programs  had  fast 
paced  schedules,  their  early  flight  test  programs  were  not  particularly 
successful,  and  deployment  occurred  before  the  systems  were  fully  ma¬ 
ture.  (Each  system,  though,  still  possessed  moderate-to-high  operational 
utility.) 
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Four  of  the  first  10  Minuteman  I  flight  tests  (beginning  February  1, 
1961)  were  failures  or  partial  failures.  (In  addition,  the  first  Minuteman 
I  flight  was  roughly  11  months  after  the  initial  production  decision!) 
Flights  of  the  first  five  Polaris-configured  flight  test  vehicles  (AX  series, 
beginning  September  24,  1958)  ended  in  failures.  Both  the  candidate 
Boeing  and  General  Dynamics  long-range  ALCMs  suffered  four  early 
terminations  each  in  their  10  missile  FSD  flyoff  (beginning  August  3, 
1979  and  July  17,  1979,  respectively). 

Consequently,  four  or  more  of  the  first  10  flight  tests  were  failures  or 
partial  failures  for  each  program.  In  effect,  the  program  managers  and 
cognizant  DoD  personnel  had  considerable  confidence  in  each  weapon 
system  and  protected  them  from  adverse  external  reactions.  Any  DoD 
program  today  having  a  succe.ss  factor  of  60  percent  in  its  first  10  flight 
tests,  regardless  of  its  flight  test  record  afterwards,  would  almost  cer¬ 
tainly  encounter  considerable  monitoring  from  external  organizations 
(e.g.,  the  Congress),  that  might  adversely  impact  program  cost,  perfor¬ 
mance,  and/or  schedule,  if  not  cause  program  termination. 

Summary  of  Some  Common  Program  Characteristics 

At  this  point,  I  will  summarize  some  of  the  key  characteristics  of  the 
Minuteman  I,  Polaris  A-1,  and  ALCM  programs  discussed  so  far. 

First,  each  was  a  program  of  the  highest  national  importance  and  held 
a  BRICK-BAT  DX  priority  rating.  Second,  each  program  was  not  wholly 
insulated  from  externally  induced  program  impacts  during  the  develop¬ 
ment  phase.  Similarly,  the  programs  were  impacted  by  service  and  DoD- 
level  organizational  and  policy  changes.  Third,  each  program  had  only 
modest  to  moderate  support  until  mid-way  in  the  development  phase. 
To  a  great  degree,  factors  external  to  the  Minuteman  I.  Polaris,  and 
ALCM  program  offices  led  to  increased  program  support.  Fourth,  while 
each  program  received  the  necessary  funding,  potential  funding  short¬ 
falls  occurred  that  produced  moderate  program  turbulence  in  the  short 
run.  In  addition,  the  accelerated  program  schedules  led  to  moderate 
development  phase  cost  growth  in  the  Minuteman  I  and  ALCM  pro¬ 
grams.  Fifth,  each  program  required  a  moderate  technology  advance 
across  the  entire  .system,  with  a  considerable  advance  in  the  state  of  the 
art  required  for  only  a  few  subsystems.  Each  program  was  the  benefi¬ 
ciary  of  important  advances  made  from  prior  technology  or  operational 
programs.  Sixth,  the  initial  thrust/weight  ratio  associated  with  engine 
propulsion,  coupled  with  missile  weight,  proved  to  be  optimistic.  Sev¬ 
enth,  parallel  R&D  and  simultaneous  exploration  of  several  alternatives 
were  used  extensively  as  risk  reduction  measures  for  a  number  of  the 
key  technology  areas.  Eighth,  the  development  phase  flight  test  pro- 
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grams  were  not  particularly  successful.  In  today’s  acquisition  environ¬ 
ment,  this  would  likely  lead  to  program  schedule  slippage,  if  not  termi¬ 
nation. 

CAN  THE  LESSONS  LEARNED  BE  TRANSFERRED 
TO  OTHER  DoD  PROGRAMS? 

Given  the  common  characteristics  of  the  three  programs,  I  will  now  briefly 
discuss  whether  or  not  lessons  learned  from  the  Minuteman  I,  Polaris  A-1, 
and  ALCM  programs  can  be  transferred  to  other  DoD  programs. 

First,  the  success  of  the  Minuteman  1,  Polaris  A-1,  and  ALCM  pro¬ 
grams  was  largely  due  to  an  unu.sual  convergence  of  technological  op¬ 
portunities  and  a  consensus  on  national  needs. 

The  ballistic  and  cruise  missile  research  programs  had  already  made 
considerable  strides  in  the  technology  associated  with  key  subsystems 
for  the  Minuteman  1,  Polaris  A-1,  and  ALCM  prior  to  the  FSD  (equiva¬ 
lent)  program  phase.  Hence,  the  underlying  technologies  needed  for 
these  missiles  were  already  on  the  necessary  path  to  yield  viable  subsystems. 

In  addition,  there  was  a  clear  consensus  on  national  needs  associated 
with  the  perceived  Soviet  ballistic  missile  gap,  coupled  with  the  Sputnik 
program,  that  assisted  both  the  Minuteman  I  and  Polaris  A-1  beginning 
in  late  1957  (Perry). 

As  noted  by  Robert  Perry,  the  Minuteman  1  and  Polaris  A-1  accelera¬ 
tion  decisions  were: 

.  .  .  made  feasible  by  the  pace  of  technology — which  cer¬ 
tainly  had  been  more  rapid  for  ballistic  missiles  than  for 
weapons  contemporary  with  them.  But  the  decisions  prob¬ 
ably  would  not  have  been  made  as  they  were  if  the  Soviet 
Union  had  not  provided  first  rate  motivation:  an  unmis¬ 
takable  threat. 

Similarly,  upgraded  Soviet  air  defenses  made  the  role  of  a  long-range 
ALCM  relatively  more  important  than  a  short-range  ALCM  coupled 
with  the  B-IA  penetrating  bomber,  that  had  less  extensive  stealth  char¬ 
acteristics  than  either  ALCM. 

The  breakthrough  that  permitted  the  rapid,  extensive  deployment  of 
these  missiles  was  far  more  political  than  technological.  The  political 
consensus  permitted  virtually  unrestricted  program  funding,  although 
moderate  short  run  funding  turbulence  did  exist,  and  diminished  the 
influence  of  program  critics. 

In  effect,  the  convergence  of  technological  opportunity  coupled  with 
a  consensus  of  national  needs  has  rarelv  existed  to  the  extent  that  it  did 
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for  the  Minuteman  I,  Polaris,  and  ALCM  programs  except  for  the  Man¬ 
hattan  project,  the  Apollo  project,  and  perhaps  a  few  others.  These 
were  both  fortunate  and  special  programs  in  a  number  of  ways,  and 
although  the  technological  advances  made  in  each  program  were  note¬ 
worthy,  the  political  environment,  defined  through  a  consensus  of  na¬ 
tional  needs,  that  existed  in  each  case  played  a  dominant  role  in  the 
development,  deployment,  and  success  of  these  systems.  Without  both 
technological  opportunities  and  a  consensus  on  national  needs,  it  is  un¬ 
likely  that  many  military  programs  run  with  a  conventional  acquisition 
strategy  can  achieve  the  accelerated  (development)  schedules  typically 
associated  with  successful  commercial  programs. 

In  some  cases,  a  consensus  of  service  needs  may  be  sufficient,  while 
for  other  programs,  a  consensus  of  DoD,  government  (including  the 
Congress),  or  even  national  needs  may  be  necessary  to  insure  a  {success¬ 
ful  program  development  and  deployment  in  terms  of  the  required  per¬ 
formance  and/or  schedule. 

As  Harvey  Sapolsky  stated  in  1972,  “Most  government  undertakings, 
whether  civilian  or  military,  are  apparently  not  the  beneficiaries  of  a  conver¬ 
gence  between  technological  opportunities  and  political  consensus.” 

Second,  of  the  eight  common  Minuteman  I,  Polaris  A-1,  and  ALCM 
program  characteristics  previously  identified,  only  two  are  both  desir¬ 
able  and  somewhat  within  the  control  of  the  government  program  office 
director.  These  include  the  need  for  only  a  moderate  average  advance  in 
the  required  technology  state  of  the  art  and  the  use  of  parallel  R&D  for 
risk  reduction  (assuming  adequate  funding  is  available). 

The  other  six  items  previously  mentioned  either  require  actions  exter¬ 
na!  to  the  government  program  office  to  implement  (e.g.,  receiving  a 
BRICK-BAT  DX  priority  rating  and  service,  DoD-levcl  organizational 
and  policy  changes,  and  receiving  necessary'  funding),  or  represent  ac¬ 
tions  with  potentially  undesirable  effects  to  the  program  (e.g.,  limited 
early  support,  optimistic  missile  thrust/weight  ratio,  and  a  not  particu¬ 
larly  successful  development  phase  flight  test  program). 

Consequently,  applying  lessons  learned  from  the  Minuteman  I,  Po¬ 
laris  A-1,  and  ALCM  programs  to  a  new  military  development  program 
may  assist  the  program  to  some  extent  (e.g.,  parallel  R&D  for  risk  re¬ 
duction  if  funding  is  available).  However,  there  is  no  guarantee  that  the 
program  will  achieve  the  same  degree  of  success  as  in  the  Minuteman  I, 
Polaris  A-1,  and  ALCM  programs.  The  level  of  success  achieved  by 
these  programs  was  to  a  great  extent  determined  by  both  necessary 
technological  opportunities  and  a  national  consensus  of  needs,  which 
were  external  to  the  government  program  offices  and  even  the  DoD 
itself. 
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rhe  ability  to  quantify  risk  is  essential  to  the  processes  of  budgeting 
and  scheduling.  During  the  process  of  hiring  to  complete  specified 
tasks,  customers  must  be  able  to  verify  contractor  estimates  and  to 
make  sound  judgments  on  the  risks  of  cost  overruns  and  time  delays.  The 
following  questions  are  central  to  this  paper:  Do  developers  with  little  expe¬ 
rience  overestimate  or  underestimate  the  complexity^  of  the  task  because  of 
their  experience,  the  assumptions  they  make,  the  models  they  select,  and 
how  they  define  the  model  limits?  What  are  the  sources  of  risk  associated 
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with  project  cost  estimation?  How  can  such  risk  be  quantified?  This  article 
proposes  a  systematic  acquisition  process  aimed  at  assessing  and  managing 
the  risks  of  cost  overruns  and  time  delays  associated  with  software  develop¬ 
ment.  We  propose  an  acquisition  process  of  four  phases  grounded  on  three 
basic  premises:  (a)  Any  single-value  estimate  of  cost  or  completion  time  is 
inadequate  to  capture  and  represent  the  variability  and  uncertainty  associ¬ 
ated  with  cost  and  schedule.  Probabilistic  quantification  is  advocated,  us¬ 
ing,  in  this  paper,  the  fractile  method  and  triangular  distribution,  (b)  The 
common  expected  value,  when  used  as  a  measure  of  risk,  is  inadequate; 
further,  if  used  as  the  sole  measure  of  risk,  it  may  lead  to  inaccurate  results. 
The  conditional  expected  value  of  risk  of  extreme  events  is  adopted  to 
supplement  and  complement  the  common  unconditional  expected  value, 
(c)  Probing  the  sources  of  risks  and  uncertainties  associated  with  cost  over¬ 
runs  and  time  delays  in  software  development  is  essential  for  the  ultimate 
management  of  technical  and  nontechnical  risks.  This  article  is  based  on  a 
technical  report  published  by  the  Software  Engineering  Institute  (Haimes 
and  Chittister,  1993). 


PREFACE 

The  software  development  community  has  not  been  able  to  agree  upon 
a  set  of  measures  to  define  the  basic  building  blocks  that  can  be  used  to 
generate  cost  and  schedule  estimates.  For  example,  in  most  other  engi¬ 
neering  fields,  cost  estimates  are  based  on  basic  measures.  Examples 
include  BTUs,  PSIs,  length  of  experience,  complexity  of  the  require¬ 
ments,  software  language  to  be  used,  and  estimated  number  of  lines  of 
code.  The  relationships  among  these  factors  and  the  cost  or  schedule 
estimate  are  not  always  clear,  and  this  raises  some  questions  as  to  the 
validity  of  the  estimates  in  any  particular  case. 

The  following  quotations  excerpted  from  Innovative  Contracting  Prac¬ 
tices:  Transportation  Research  Circular  No.  386,  published  by  the  Na¬ 
tional  Research  Council  (December  1991)  highlight  the  current  dismal 
state  of  contracting  practices: 

•  Innovative  contracting  techniques  have  been  developed  more  in 
foreign  countries  than  in  the  United  States. 

•  Unfortunately,  the  lowest  initial  cost  may  not  result  in  the  lowest 
overall  cost. 

•  In  fact,  current  contracting  practices  provide  little  incentive  for 
industry  to  be  innovative. 
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•  Agencies  should  develop  contractor  responsibility  tests  that  reflect 
quality  and  performance  factors;  these  tests  should  be  examined 
and  possible  modifications  developed. 

•  Indeed,  the  ability  to  assess  quality  and  performance  are  directly 
related  to  the  ability  to  assess  risk. 

•  From  a  Summary  of  Questionnaire  Findings: 

This  [pre-bid  conferences]  concept  was  the  most  popular,  receiv¬ 
ing  a  positive  response  from  more  than  85%  of  the  states  partici¬ 
pating  in  the  survey.  Better  understanding  of  the  scope  of  work, 
reduction  in  unanticipated  construction  conflicts,  plan  revisions, 
and  other  value  engineering  benefits  can  result  from  such  con¬ 
ferences.  Specialty  jobs,  especially  fast-track  projects,  are  most 
appropriate  for  this  process. 

•  Risk  management  and  assurance.  End-result  specifications  and  a 
determination  of  QA  enter  into  this  issue.  Although  not  currently 
being  practiced,  many  agencies  are  considering  this  concept  for 
future  application. 

The  questionnaire  indicated  that  innovation  has  intensified  in  selected 
topic  areas.  Many  agencies  are  implementing  quality  assurance-quality 
control  (QA-QC)  philosophies,  contractor  surveying,  value  engineering, 
off-peak  time  incentives,  alternative  bidding  on  structures,  and  other 
concepts.  Additionally,  cost-saving  and  profitable  concepts  are  being 
considered  for  future  use  and  further  development.  On  the  other  hand, 
many  agencies  expressed  interest  in  receiving  guidelines  on  other  con¬ 
cepts  that  were  less  understood. 

INTRODUCTION 

Three  major  classes  of  likely  adverse  consequences  arc  prevalent  in 
software  development:  risk  of  cost  overrun,  risk  of  time  delay  in  the 
completion  schedule,  and  risk  of  not  meeting  performance  specifica¬ 
tions.  Here  risk  is  defined  as  a  measure  of  the  probability  and  severity  of 
adverse  effects  (Lowrance  1976).  The  first  two  risks,  cost  overrun  and 
time  delay,  are  nontechnical  risks  and  the  third,  performance  specifica¬ 
tions,  is  software  technical  risk;  more  precise  definitions  are  found  in 
Chittister  and  Haimes  (1993).  The  focus  of  this  paper  is  on  the  quantifi¬ 
cation  (assessment)  and  management  of  software  nontechnical  risks, 
such  as  cost  overruns  and  time  delays. 

The  more  central  the  role  that  software  plays  in  overall  system  inte- 
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gration  and  coordination,  the  more  likely  the  impact  of  delivery  delay 
and/or  of  major  cost  overruns.  A  series  of  auditing  studies  conducted  by 
the  General  Accounting  Office  (GAO)  in  1992  reveal  almost  an  across- 
the-board  epidemic  of  cost  overruns  and  time  delays  in  meeting  comple¬ 
tion  schedules  associated  with  software  development  for  seleeted  gov¬ 
ernment-sponsored  projects.  A  case  in  point  is  the  C-17  aircraft,  cited  in 
the  previously  mentioned  GAO  report,  which  experienced  a  major  cost 
overrun  and  delivery  delay. 

In  spite  of  the  efforts  made  by  some  of  the  Source  Selection  Authori¬ 
ties  (SSAs)  and  by  their  respective  boards  in  selecting  contractors,  the 
Department  of  Defense  (DoD)  has  still  had  serious  software  delays. 
Indeed,  an  SSA  conducts  a  thorough  .search,  examining,  among  other 
factors,  the  organizational  capabilities  of  the  contractor  by  evaluating 
performance  histor>'.  In  some  cases  a  set  of  Key  Practice  Areas  (KPAs) 
are  examined,  such  as  the  processes  of  formal  cost  e.stimation  and  pro¬ 
gram  management,  as  well  as  metrics  for  evaluating  various  performance 
criteria. 

The  Software  Engineering  Institute  (SEI)  at  Carnegie  Mellon  Uni¬ 
versity  has  al.so  developed  a  methodology  known  as  Software  Capability 
Evaluation  (SCE)  that  (Humphrey  &  Sweet  1987)  used  to  assess  the 
software  engineering  capability  of  contractors.  The  SCE  seeks  to  answer 
the  question:  Can  the  organization  build  the  product  correctly?  It  does 
so  by  considering  three  separate  aspects  of  the  contractor’s  expertise: 

•  organization  and  resource  management; 

•  the  software  engineering  proce.ss  and  its  management;  and 

•  available  tools  and  technology. 

Another  tool,  a  risk  taxonomy,  al.so  developed  at  SEI,  addresses  the 
sources  of  software  technical  risk  and  attempts  to  answer  the  question:  “Is 
the  organization  building  the  right  product?”  (Carr  et  al.  1993).  The  SCE 
and  the  taxonomy,  then,  offer  methods  of  assessing  organizational  pro¬ 
cesses  and  software  technical  risks;  this  article  presents,  on  the  other  hand, 
a  process  for  quantifying  the  risks  of  project  cost  and  schedule  overruns. 

OVERVIEW  OF  THE  CONCEPTUAL  FRAMEWORK 

In  this  article,  we  present  a  methodological  framework  for  selecting  a 
contractor  that  can  assist  the  customer  in  minimizing  the  risks  of  project 
cost  overruns  and  schedule  delays.  Although  factors  other  than  the  se¬ 
lection  of  contractor(s)  may  decisively  affect  software  technical  and  non- 
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technical  risks,  they  are  treated  here  only  as  a  general  background.  See 
Chittister  and  Haimes  (1993,  1994)  for  a  more  in-depth  discussion  of 
these  factors. 

The  process  of  selecting  contractors  is  by  itself  quite  complex.  The 
process  is  driven  by  legal,  organizational,  technical,  financial,  and  other 
considerations — all  of  which  serve  as  sources  of  risk.  Because  the  world 
within  which  software  engineering  developed  is  nondeterministic  and 
the  central  tendency  measure  of  random  events  conceals  vital  and  criti¬ 
cal  information  about  these  random  events,  special  attention  is  focused 
on  the  variance  of  these  events  and  on  their  extremes.  Two  approaches — 
the  fractilc  method  and  triangular  distribution — are  adopted  here  to 
quantify  the  probabilities  of  project  cost  overrun  and  delay  in  comple¬ 
tion  schedule.  To  capture  the  range  of  variation  and  the  extremes  of 
these  probabilities,  conditional  expected  values  of  extreme  events  are 
calculated  using  the  partitioned  multiobjcctivc  risk  method  (PMRM) 
(Asbcck  &  Haimes  1984)  to  supplement  the  common  expected  value  of 
software  nontechnical  risk.  To  accomplish  this  objective,  the  fractile 
method,  triangular  distribution,  and  the  PMRM  are  introduced.  Ex¬ 
amples  are  included  to  clarify  the  appropriate  application  of  these  meth¬ 
ods  and  to  demonstrate  their  utility. 

Figure  1  represents  the  thinking  of  the  methodological  framework. 
The  framework  can  be  viewed  with  four  major  phases.  The  purpose  of 
Phase  I  is  to  quantify  the  variances  in  the  contractor’s  cost  and  schedule 
estimates  by  constructing  probability  density  functions  (PDFs)  through 
triangular  distributions,  the  fractilc  method,  or  through  any  other  meth¬ 
ods  that  seem  suitable  to  the  contractor.  Extreme  events  are  assessed 
from  these  PDFs.  In  Phase  II,  using  the  SEI  Taxonomy,  interviews,  and 
the  PMRM,  the  sources  of  risks  and  uncertainties  associated  with  each 
contractor  are  probed  and  evaluated.  The  assumptions  and  premises, 
which  provide  the  basis  for  generating  the  variances  in  the  contractor’s 
estimates,  are  identified  and  evaluated;  and  the  conditional  expected 
value  of  risk  of  extreme  cost  overruns  and  time  delays  are  constructed 
and  evaluated.  Phase  III  analyzes,  ranks,  filters,  and  compares  the  sig¬ 
nificance,  interpretation,  and  validity  of  each  contractor's  assumptions 
and  premises.  And  the  probabilities  of  technical  and  nontechnical  risks 
are  assessed.  In  executing  Phase  III.  three  tools  and  methodologies  are 
used:  (1)  independent  verification  and  validation  team;  (2)  the  risk  rank¬ 
ing  and  filtering  method;  and  (3)  comparative  analysis.  In  the  final  phase, 
Phase  IV,  conclusions  are  drawn  on  the  basis  of  all  the  previously  gener¬ 
ated  evidence,  including  the  opinions  of  expert  judgment.  The  ultimate 
objective  of  the  methodological  approach  is  to  minimize  the  following 
three  objectives  or  indices  of  performance: 


Acquisition  Review  Quarterly 


Spring  1995  -  125 


An  Acquisition  Process  for  the  Management  of  Nontechnical  Risks 
Associated  With  Software  Development 


Um:  — » 

•  Triangular 

OiatritHihon 

•  Fraetha  Mathod 


•  Contiruet  POFi 
■  Aiaaai  extrama  avanti 


(Ha; 

•  Taxonomy 


•  Pioba  tha  aeureaa  el  rttka  and  imcanainiuaa 

•  MantHy  and  avakiata  tna  ataumpoont  that  have 

panaratad  tha  vartaneat  lor  aach  Oidding  contractor 
\«^onttrvct  rtah  ol  axirama  avanta _ ^ 


Uaar 

•  Mapandant  Vartheatlon  and 

Validation  (IVV)  Taam 

•  Rlik  Ranking  and  Ritaring  (RRF) 

•  ComparaUva  Anaiyats 


r 

PHASE  IV 

•  MulPob^ettve 

^  PHASE  III 

•  Draw  eonelualcina  baaad  on  iho  avldanca 

TfAdeOff 

Aneiyela 

•  Anafyx*  and  eemparo  the  algnsicAned  and 

(MippiemanMd  by  export  judgment) 

J 

vAMity  ol  Ihoaa  aaaumpUont  lor  tha  UkaH- 
hoed  ol  Mchnlcal  and  rtontacnrVcal  rtafci 

Figure  1.  Proposed  Acquisition  Process 


Minimize 


risk  of  project  cost  overrun; 

risk  of  project  completion  time  delay; 

risk  of  not  meeting  performance  criteria. 


Clearly,  multiobjective  trade-off  analysis,  using  for  example,  the  sur¬ 
rogate  worth  trade-off  (SWT)  method,  needs  to  be  conducted  where  all 
costs  and  risks  are  kept  and  trade  off  in  their  units. 

The  objective  of  this  article  is  to  develop  scientifically  sound  and 
pragmatic  answers  to  some  of  the  lingering  problems  and  questions 
concerning  assessment  and  management  of  risks  of  cost  overruns  and 
time  delays  associated  with  software  engineering  development. 

It  is  constructive  to  discuss  the  four-phase  acquisition  process  in  more 
detail: 


1.  Phase  I  will  be  demonstrated  through  the  construction  of  the  prob¬ 
ability  density  functions  (using  the  fractile  method  and  triangular 
distribution)  and  through  the  assessment  of  extreme  events  (using 
the  partitioned  multiobjective  risk  method)  by  calculating  the  con¬ 
ditional  expected  value  of  extreme  events  to  supplement  the  com¬ 
mon  unconditional  expected  value  of  cost  overrun. 

2.  Phase  II,  through  the  use  of  the  Taxonomy-Based  Questionnaire, 
interviews,  and  the  quantification  of  risk  of  extreme  events,  pro¬ 
vides  a  mechanism  to  probe  the  sources  of  risks  and  uncertainties, 
identify  and  evaluate  the  assumptions  that  have  generated  the  vari¬ 
ances  for  each  bidding  contractor,  and  construct  the  conditional 
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expected  value  of  risk  of  extreme  events,  f4(«).  The  following  dis¬ 
cussion  will  focus  on  probing  the  sources  of  extreme  events  and 
the  contractor’s  attitude: 

Extreme  Events  —  The  shape  of  the  probability  density  function,  and 
particularly  the  behavior  of  the  tail  of  the  distribution,  markedly  influ¬ 
ence  the  conditional  expected  value  of  extreme  events.  To  demonstrate 
the  effect  of  the  tail  of  the  distribution  on  projected  cost  overruns  or 
time  delays,  all  three  examples  have  at  least  one  project  cost  estimate 
with  a  long  tail  (i.e.,  a  major  cost  overrun  albeit  with  a  relatively  low 
probability). 

Most  customers  are  concerned  with  major  cost  overruns  and  time 
delays  of  any  magnitude.  In  other  words,  most  customers  want  to  pre¬ 
vent  disastrous  events  that  are  beyond  point  b  in  Figure  2.  The  region  to 
the  left  of  b  (cost  overruns  that  would  not  exceed  10-15%)  is  commonly 
represented  by  the  expected  value  measure  of  risk,  f5(*),  whereas  the 
region  to  the  right  of  b  is  captured  by  the  conditional  expected  value  of 
risk  of  extreme  events,  f4(«).  With  the  help  of  the  Taxonomy-Based 
Questionnaire  we  can  probe  the  sources  of  uncertainties  and  variabili¬ 
ties  leading  to  f5(«)  and  f4(«).  Indeed,  the  ultimate  efficacy  of  risk  assess¬ 
ment  is  its  management  through  early  identification,  quantification,  and 
prevention.  Such  a  probe  provides  insights  into  the  contractor’s  assump¬ 
tions  about  what  can  go  wrong  in  a  severe  way  that  might  cause  the  risk 


Low  and  Moderate  Severity 

Medium  and  High  Exceedance  Probability 


Figure  2.  Mapping  the  Probability  Partitioning  Onto  the  Damage  Axis 
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of  extreme  cost  overrun  or  time  delay  to  be  catastrophic. 

Contractor’s  Attitude  —  The  Taxonomy-Based  Questionnaire,  along 
with  the  measurements  of  risk  of  cost  overruns  and  time  delays  through 
the  conditional  expected  value  of  risk  of  extreme  events  and  the  uncon¬ 
ditional  (common)  expected  value  of  risk  should  explain  not  only  the 
contractor's  technical,  financial,  and  other  managerial  assumptions  and 
premises,  but  also  the  contractor’s  attitude  toward  risk.  When  a  contractor’s 
projection  of  lowest,  most  likely,  and  highest  project  costs  falls,  for  example, 
in  a  close  range,  there  arc  several  possible  explanations: 

1.  The  contractor  is  a  risk  seeker  (a  risk-averse  contractor  would 
have  projected  a  much  wider  spread  in  the  lowest,  highest,  and 
most  likely  project  cost). 

2.  The  contractor  is  very  knowledgeable  and  thus  has  confidence  in 
the  tight  projections. 

3.  The  contractor  is  ignorant  of  the  major  technical  details  and  com¬ 
plexity  of  the  project’s  specifications;  thus,  inherent  uncertainties 
and  variabilities  associated  with  the  project  have  been  overlooked. 
Otherwise,  the  contractor  would  have  projected  a  wider  spread 
between  the  most  likely  and  highest  cost  projections. 

The  Taxonomy  not  only  constitutes  an  important  instrument  with  which 
to  discover  the  reasons  for  the  uncertainties  and  variabilities  associated 
with  the  contractor’s  projections,  it  also  provides  a  mechanism  that  al¬ 
lows  the  customer  to  a.ssess  the  validity  and  soundness  of  the  contractor’s 
assumptions,  indeed,  the  Taxonomy-Based  Questionnaire,  which  is  sys¬ 
tematic,  structured,  and  repeatable,  is  an  invaluable  process  with  which 
the  customer  can  elicit  answers  to  the  rca.sons  for  the  contractors’  vari¬ 
abilities.  The  accumulated  assumptions  on  each  contractor  are  then  com¬ 
pared  and  analyzed. 

In  Phase  III,  an  analysis  and  compari.son  of  the  significance  and  valid¬ 
ity  of  the  contractor’s  assumptions  about  technical  and  nontechnical 
risks  is  conducted.  This  is  accomplished  by  an  Independent  Verification 
and  Validation  (IVV)  team,  the  Risk  Ranking  and  Filtering  (RRF) 
method,  and  other  comparative  analysis  methods.  In  comparing  assump¬ 
tions,  a  number  of  issues  must  be  addressed;  stability  and  precedence  of 
the  requirements;  need  for  research  about  solutions;  politics  and  stabil¬ 
ity  of  funding:  overall  knowledge  or  the  lack  thereof;  level  of  experience 
of  key  personnel;  maturity  of  technology;  and  maturity  of  the  organiza¬ 
tion. 
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In  making  these  comparisons,  the  customer  will  look  at  the  reasons 
for  the  assumptions  and  whether  they  are  based  on  knowledge  or  naivete, 
and  whether  the  contractor  has  a  conservative/risk-averse  or  liberal/risk¬ 
seeking  attitude.  These  issues  are  highlighted  in  the  example  problem  in 
subsequent  discussions.  Contractor  A  is  projecting  a  50%  cost  overrun 
as  the  worst  case,  while  Contractor  B  is  projecting  “only”  a  40%  cost 
overrun  as  the  worst  case.  Is  Contractor  A  more  knowledgeable  or  more 
conservative  than  Contractor  B?  Or  does  the  reason  for  this  difference 
lie  elsewhere?  Is  Contractor  A  risk-averse  whle  Contractor  B  is  risk 
seeking?  The  information  generated  by  the  I VV  team,  the  RRF  method, 
and  through  other  comparative  analysis  tools  will  be  subjected  to  the 
expert  judgment  of  the  customer’s  team,  leading  to  Phase  IV  of  the 
proposed  acquisition  process. 

Phase  IV  is  the  completion  step  where  conclusions  are  drawn  based 
on  the  accumulated  evidence.  Expert  judgment  is  used  in  conjunction 
with  multiobjective  trade-off  analysis  methods,  such  as  the  surrogate 
worth  trade-off  (SWT)  method  (Haimes  &  Hall  1974).  Adopting  the 
systemic  proposed  acquisition  process  should  markedly  reduce  the  like¬ 
lihood  of  major  and  catastrophic  technical  and  nontechnical  risks. 

CRITICAL  FACTORS  THAT  AFFECT 
SOFTWARE  NONTECHNICAL  RISK 

The  proposed  methodological  framework  for  the  quantification  and 
management  of  software  nontechnical  risk — the  risk  of  cost  overrun  and 


Figure  3.  Graphical  CDF  for  Project  Cost  Increase  for  Contractor  A 
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time  delay  associated  with  software  development — is  grounded  on  the 
premise  that  such  management  must  be  holistically  based.  A  holistic 
approach  requires  complete  accounting  of  all  important  and  relevant 
forces  that  drive  the  dynamics  of  cost  overrun  and  time  delay.  Although 
a  holistic  view  is  advocated,  introduced,  and  discussed  in  this  article, 
only  limited  aspects  are  ultimately  quantified.  Indeed,  only  a  series  of 
articles  would  do  justice  to  the  quantification  of  risks  associated  with  all 
factors  that  embody  software  nontechnical  risk.  By  their  nature,  quanti¬ 
fication  and  management  of  software  nontechnical  risk  (and  to  a  large 
extent  software  technical  risk)  involve  the  customer  and  the  shadow 
client  (the  U.S.  Congress,  in  the  case  of  DoD)  and  the  contractor(s). 
Organizational  interface  between  the  customer  and  the  contractor(s), 
the  state  of  technology  and  know-how,  the  complexity  of  the  specifica¬ 
tion  requirements,  add-on  modifications  and  refinements,  the  availabil¬ 
ity  of  appropriate  resources,  and  the  models  used  for  project  cost  esti¬ 
mation  and  schedule  projections  are  major  considerations. 

Since  each  element  is  in  itself  a  complex  entity  with  diverse  dimen¬ 
sions,  it  is  essential  to  recognize  which  characteristics  of  each  compo¬ 
nent  contribute  to  software  nontechnical  risk.  Only  by  understanding 
the  sources  of  risk  can  it  ultimately  be  prevented  and  managed. 

The  Customer 

The  term  customer  is  a  misnomer  becau.se  it  connotes  a  singular  entity. 
Yet.  in  most  large-scale  software  engineering  systems,  such  as  DoD  sys¬ 
tems,  projects  are  initiated,  advocated,  nourished,  and  supported  by 
multiple  constituencies  with  some  common,  but  often  different,  goals 
and  objectives.  Furthermore,  for  DoD  projects,  there  is  also  the  shadow 
customer — the  U.S.  Congress,  itself  influenced  by  various  lobbyists,  power 
brokers,  and  stakeholders.  The  existence  and  influence  of  this  multiplic¬ 
ity  of  clients  on  the  ultimate  resources  for  the  development  of  software 
engineering  constitute  a  critical  source  of  software  risk.  It  is  not  uncom¬ 
mon  for  a  pressure  group  to  affect  either  the  design  specifications  and/ 
or  the  resources  allocated  for  a  specific  DoD  project,  and  thus  to  have 
an  impact  on  its  final  cost  and  its  completion  time.  The  “organizational 
maturity"  level  of  the  client  is  another  factor  that  influences  software 
nontechnical  risk.  A  client  that  possesses  internal  capabilities  to  com¬ 
municate  with  the  contractor(s)  on  both  the  technical  and  nontechnical 
levels  is  more  likely  to  have  a  better  understanding  and  therefore  man¬ 
agement  of  software  nontechnical  risk.  This  attribute  will  become  more 
evident  in  this  article  as  specific  quantitative  information  on  the  vari¬ 
ances  of  cost  and  schedule  is  solicited  in  the  proposed  methodological 
framework. 
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The  Contractor(s) 

Elaborate  procedures  and  protocols  describing  contractor  selection  for 
the  development  of  software  engineering  have  been  designed  and  are 
being  employed  by  government  agencies  and  corporations.  A  commonly 
accepted  axiomatic  premise  is  that  the  organizational  maturity  level  of 
the  contractor  and  the  experience,  expertise,  and  qualifications  of  its 
staff  markedly  affect  the  management  of  software  technical  and  non¬ 
technical  risks. 

The  Interface  Between  the  Customer  and  the  Contractor(s) 

One  of  the  dominant  factors  in  initiating  both  technical  and  nontechni¬ 
cal  risks  can  be  traced  to  the  organizational  interface  between  the  cus¬ 
tomer  and  the  contractor(s).  Adequate  and  appropriate  communication 
between  the  two  parties,  and  the  understanding  and  the  appreciation  of 
each  other’s  role  throughout  the  life  cycle  of  the  software  development 
process  are  imperatives  to  the  prevention  and/or  control  of  potential  risks. 

The  State  of  Technology  and  Know-how 

Although  many  consider  the  contractor’s  access  to  knowledge  and  the 
access  to  appropriate  technology  to  be  major  factors  in  controlling  soft¬ 
ware  technical  risk,  these  factors  also  impact  software  nontechnical  risk. 
In  particular,  the  lack  of  access  to  appropriate  technology  or  a  deficient 
know-how  in  the  contractor’s  staff  is  likely  to  cause  a  measurable  time 
delay  in  the  completion  of  a  project  and  is  also  likely  to  cause  cost 
overrun. 

The  Complexity  of  the  Specirication  Requirements 

The  more  unprecedented  the  client’s  project  specifications  in  terms  of 
advanced  and  emerging  technology,  the  higher  the  risk  of  time  delay  in 
its  completion  and  of  its  cost  overrun.  Most  systems  developed  by  the 
DoD  are  advancing  the  state  of  the  art  in  some  field  of  technology,  e.g., 
software  development,  stealth,  propulsion,  and  satellites.  The  require¬ 
ments  in  these  fields  are  necessarily  complex  since  the  parameters  are 
constrained  by  the  task  and  are  frequently  subject  to  modifications  be¬ 
cause  of  changing  technology. 

The  Add-on  ModiFications  and  Refinements 

Add-on  modifications  and  refinements  are  viewed  by  many  as  an  Achil¬ 
les’  heel  in  terms  of  software  nontechnical  risk.  Although  these  add-on 
modifications  are  often  associated  with  software  nontechnical  risk,  they 
also  constitute  a  critical  source  of  software  technical  risk.  This  is  be¬ 
cause  not  all  modifications  are  appropriately  related  to  and  checked 
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against  the  original  design  to  ensure  ultimate  compatibility  and  har¬ 
mony.  Very  large  and  complex  systems  are  difficult  to  manage.  Systems 
are  now  developed  by  multiple  companies,  each  having  its  own  area  of 
expertise,  and  changes  often  ripple  through  the  entire  system.  A  wide 
range  of  factors  may  cause  mid-course  modifications;  however,  three 
categories  of  causes  emerge  from  this  range: 

a.  Threat  or  Need  Change:  When  a  new  threat  is  projected  or  a  new 
need  is  contemplated. 

b.  Improved  New  Technology:  When  a  new  technology  provides  im¬ 
proved  performance  or  quality,  such  as  a  new  sensor. 

c.  Obsolete  Technology  Replacement:  When  the  pre-selected  tech¬ 
nology  becomes  obsolete  before  the  contract  has  even  begun  or 
been  completed. 

The  Availability  of  Appropriate  Resources 

One  open  secret  in  government  procurement  and  occasionally  in  the 
private  sector  is  the  level  of  pre-allocated  funds  for  a  specific  project. 
The  competitive  zeal  of  contractors  often  outweighs  the  technical  judg¬ 
ment  of  their  professional  staff;  the  outcome  is  a  bid  that  is  close  to  the 
pre-allocated  funds  by  the  client  even  though  it  is  clear  to  the  bidder 
that  the  job  with  its  specification  requirements  cannot  be  delivered  at 
that  level  of  funding.  This  not-uncommon  phenomenon  is  dramatically 
illustrated  in  numerous  documented  examples  by  Hedrick  Smith  (1988). 

The  standard  technique  is  to  get  a  project  started  by 
having  the  prime  contractor  give  a  low  initial  cost  esti¬ 
mate  to  make  it  seem  affordable  and  wait  to  add  fancy 
electronics. and  other  gadgets  much  later  through  engi¬ 
neering  “change  orders,”  which  jack  up  the  price  and  the 
profits.  Anyone  who  has  been  through  building  or  re¬ 
modeling  a  house  knows  the  problem.  “This  is  called  the 
buy-in  game,”  an  experienced  Senate  defense  staff  spe¬ 
cialist  confided. 

The  Models  Used  for  Project  Cost  Estimation  and  Schedule  Projection 

A  number  of  models  arc  used  to  estimate  project  cost  and  its  comple¬ 
tion  schedule.  Constructive  Cost  Model  (COCOMO)  (Boehm  1981)  and 
Program  Evaluation  and  Review  Technique  (PERT)  are  representative 
examples.  Models  can  be  potent  tools  when  they  are  well  understood. 
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are  supported  by  an  appropriate  database,  and  adhere  to  the  assump¬ 
tions  upon  which  they  are  designed  to  operate.  The  complexities  of  such 
models,  however,  often  result  in  their  misuse  and/or  invalid  interpreta¬ 
tions  of  their  results.  They  thereby  ironically  become  a  source  of  soft¬ 
ware  nontechnical  risk.  The  successful  application  of  the  proposed  meth¬ 
odological  framework,  however,  does  not  depend  on  the  specific  model 
used  by  either  the  contractor  or  the  customer  to  estimate  either  the  cost 
or  the  schedule. 

From  the  above  it  seems  that  the  sources  that  contribute  to  software 
nontechnical  risk  are  organizational  and  technical  in  nature;  they  stem 
from  failures  associated  with  the  contractor  as  well  as  the  customer.  In 
terms  of  the  contractor,  these  failures  primarily  originate  from  and  are 
functions  of  such  elements  as: 

a.  the  organizational  maturity  level; 

b.  the  process  and  procedures  followed  in  the  assessment  of  the 
project’s  cost  and  schedule; 

c.  the  level  of  honesty  exhibited  by  management  in  communicating  the 
real  cost  and  schedule  to  the  customer  (and  of  course  vice  versa); 

d.  the  extent  and  level  of  new  and  unprecedented  technology  im¬ 
posed  on  the  project; 

e.  the  level  of  experience  and  expertise  of  the  staff  engineers  in  soft¬ 
ware  engineering  in  general  and  in  the  application  domain  in  par¬ 
ticular; 

f.  the  level  of  experience  and  expertise  of  the  management  team  in 
softw'are  engineering; 

g.  the  overall  competence  of  the  team  developing  the  software; 

h.  financial  and  competitive  considerations; 

i.  immature  technology,  methods,  and  tools; 

j.  the  use  of  technology  in  new  domains; 

k.  new  combinations  of  methods  and  tools  in  new  ways  and  their  use 
in  a  new'  software  development  environment; 
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1.  requirement  modifications  causing  changes  in  the  system’s  archi¬ 
tecture. 

In  terms  of  the  customer,  the  natures  of  organizational  failures  par¬ 
tially  overlap  those  of  the  contractor’s,  but  also  have  distinctive  charac¬ 
teristics: 

a.  the  process  and  procedures  followed  in  the  assessment  of  the  project 
cost  and  schedule; 

b.  the  level  of  specificity  at  which  the  system  and  software  require¬ 
ments  are  detailed; 

c.  the  number  of  changes  and  modifications  requested  by  the  cus¬ 
tomer  during  the  software  development  process  (changes  which 
generally  introduce  many  new  errors)  are  often  not  harmonious 
with  earlier  specification  requirements; 

d.  the  commitment  of  project  management  (associated  with  the 
customer’s  organization)  to  closely  monitor  and  oversee  the  soft¬ 
ware  development  process; 

e.  the  specific  requirements  for  technology,  e.g.,  specific  compiler, 
data  base  management  systems; 

f.  the  level  of  honesty  exhibited  by  management  in  communicating 
the  real  cost  to  the  “real  client”  (e.g.,  the  Department  of  Defense 
as  a  client  and  the  U.S.  Congress  as  the  “real  client”). 

BASIS  FOR  VARIANCES  IN  COST  ESTIMATION 

Most,  if  not  all,  developers  of  large  complex  software  systems  use  cost 
models  to  estimate  their  costs.  These  models  are  based  on  a  set  of 
relationships  based  on  such  parameters  as  the  size  and  complexity  of  the 
software,  the  experience  level  of  the  software  developer,  and  the  type  of 
application  within  which  the  software  will  be  used.  Different  models 
generate  different  weights  or  levels  of  importance  for  these  parameters, 
and  not  all  models  use  the  same  parameters.  Therefore,  one  cost  model 
can  lead  to  a  radically  different  cost  estimate  than  another  just  on  the 
basis  of  which  parameters  are  used  in  the  model  and  how  they  are 
implemented.  Even  if  the  parameters  are  used  consistently,  however, 
different  developers  will  probably  not  agree  on  the  value  or  weight  of 
the  parameter  in  the  first  place.  Many  organizations,  in  fact,  consider 
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their  interpretations  of  these  parameters  to  contribute  to  their  “com¬ 
petitive  edge”  because  the  definition  affects  their  ability  to  determine 
costs  accurately.  For  example,  an  organization  that  has  experience  in 
developing  space  system  software  may  not  have  the  same  perception  of 
difficulty  when  developing  a  complex  avionics  software  system  as  would 
an  organization  that  has  significant  experience  in  that  area.  Their  under¬ 
standing  of  space  systems,  however,  will  alter  their  definition  of  the 
avionics  system  parameters.  Do  developers  with  little  experience  overes¬ 
timate  or  underestimate  the  complexity  of  the  task  because  of  how  they 
define  these  parameters?  As  stated  in  the  beginning,  these  are  questions 
central  to  this  paper:  What  are  the  sources  of  risk  associated  with  project 
cost  estimation?  How  can  such  risk  be  quantified? 

Although  creating,  maintaining,  and  updating  project  cost  estimation 
metrics  and  parameters  arc  extremely  important  for  an  organization,  it 
is  nevertheless  unlikely  that  a  future  project  will  be  similar  enough  to 
previous  projects  to  merit  directly  importing  these  metrics  or  param¬ 
eters;  such  metrics  and  parameters  may  not  be  directly  applicable  with¬ 
out  appropriate  modifications.  Indeed,  cost  estimators  are  required  to 
(and  do)  use  judgment  when  applying  these  parameters  to  a  new  project 
requirement.  Furthermore,  cost  estimation  constitutes  a  critical  area 
with  regard  to  the  sources  of  risk  for  software  development,  which  is 
without  parallel  to  other  fields.  For  example,  if  a  contractor  was  estimat¬ 
ing  the  cost  to  construct  a  building  with  50  stories,  yet  had  previously 
only  built  structures  with  a  maximum  of  10  stories,  the  contractor  would 
not  just  increase  the  estimate  five-fold.  The  contractor  would  question 
the  basic  foundations  and  relevance  of  extending  the  10-story  model  to 
the  new  structure  parameters.  However,  in  software,  it  is  not  uncommon 
to  increase  estimates  for  new  projects  by  a  factor  of  five  from  previous 
projects  of  one-fifth  the  size  and  complexity.  Many  new  systems  have 
size  estimates  of  over  1,000,000  lines  of  code  even  though  the  develop¬ 
ers  have  little  experience  with  systems  of  this  size. 

Another  example  is  in  the  use  of  commercial  off-the-shelf  (COTS) 
software.  The  original  assumption  that  a  commercial  database  manage¬ 
ment  system  (DBMS)  can  be  used  to  meet  customer  requirements  may 
change  if  the  customer  requires  features  not  supported  by  DBMS  sup¬ 
pliers.  Such  changes  may  have  serious  ramifications  for  the  cost  estimate 
depending  on  how  the  developer  plans  to  solve  the  problem.  If  the 
developer  chooses  to  subcontract  out  the  effort  and  deal  with  the  sub¬ 
contractor  in  a  way  similar  to  dealing  with  the  DBMS  vendor,  this  has 
ramifications  for  the  risk  associated  with  the  .subcontractor — an  impor- 
•  tant  subject  that  will  be  discussed  later.  The  alternative  is  for  the  devel¬ 
oper  to  undertake  the  development  of  his  or  her  own  DBMS.  This 
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requires  an  additional  set  of  assumptions,  design  parameters,  and  judg¬ 
ments  regarding  the  architecture,  size,  experience  level,  domain  know¬ 
ledge,  software  engineering  knowledge,  and  support  environment  needed 
to  develop  the  DBMS.  Each  of  these  assumptions,  parameters,  and  judg¬ 
ments  has  some  uncertainty  associated  with  it.  This  uncertainty  contrib¬ 
utes  to  the  overall  risk  in  the  cost  estimate.  If  the  developer  chooses  to 
subcontract  the  DBMS  development  to  an  outside  vendor,  then  the 
issue  for  the  contractor  is  to  understand  and  account  for  the  set  of 
assumptions  that  are  made  by  the  subcontractors  on  the  DBMS  and  on 
the  system  architecture. 

The  ability  of  the  developer  to  make  valid  assumptions  and  design 
decisions  is  usually  based  on  a  set  of  metrics;  these  metrics  can  cither  be 
based  on  current  measurements  or  on  past  performance.  Either  way, 
there  has  to  be  an  agreed-upon  set  of  measures  that  is  being  evaluated 
(such  as  the  number  of  lines  of  code  needed  to  accomplish  specified 
tasks  or  productivity  rates  in  terms  of  lines  of  code  per  hour).  The 
difficulty  with  software  development  is  that  the  community  has  not  agreed 
upon  basic  measures,  such  as  how  to  count  lines  of  code  or  how  to 
measure  productivity.  Also,  the  difficulty  with  using  performance  history 
is  that  the  systems  under  development  are  sufficiently  different  so  that 
the  history  may  not  adequately  reflect  the  new  parameters  accurately. 

THE  QUANTIFICATION  OF  SOFTWARE  NONTECHNICAL 
RISK  AND  THE  EVALUATION  OF  VARIANCES 
The  premise  of  this  article  is  that  the  manner  in  which  the  customer 
selects  a  contractor  affects  the  risk  of  cost  overruns,  time  delays,  and 
failure  to  meet  performance  specifications.  Therefore,  the  proposed 
methodological  approach  requires  the  contractor  to  provide  the  cus¬ 
tomer  with  more  than  fixed,  deterministic  values  for  the  cost  and  time 
schedule  to  deliver  software  with  prespecified  performance  requirements. 

Building  on  quantitative  probabilistic  assessment,  the  contractor  is 
asked  to  submit  either  an  estimated  variance  of  the  cost  and  time  to 
completion,  or  a  probability  density  function  (Figure  4)  for  the  pro¬ 
jected  project  cost  and  time  schedule.  (A  similar  approach  can  be  adopted 
to  quantify  software  technical  risk.)  This  variance  may  be  generated 
from,  for  example,  a  triangular  distribution,  where  the  contractor  speci¬ 
fies  the  lowest  possible  cost,  the  highest  possible  cost,  and  the  most 
likely  cost  of  the  project.  Alternatively,  the  contractor  may  choose  to 
provide  the  variance  estimate  through  the  tractile  method.  (The  con¬ 
struction  of  a  PDF  using  a  triangular  distribution  and  the  fractile  method 
will  be  subsequently  discussed.)  Similar  estimates  are  to  be  provided  for 
the  completion  schedule.  The  information  provided  by  the  contractor  con- 
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Figure  4.  Probability  Density  Function  for  Project  Cost  Increase 

for  Contractor  A 


stitutes  the  basis  for  the  generation  of  probability  density  functions  (PDFs) 
associated  with  the  project's  cost  and  its  completion  time.  Assuming  that 
the  customer  (with  the  assistance  of  a  MITRE-type  organization,  if 
needed)  is  also  capable  of  providing  (for  comparative  purposes  with  the 
clients’  variances)  basic  variance  information,  which  allows  the  genera¬ 
tion  of  the  customer’s  PDFs  for  the  projected  cost  and  completion  time, 
then  the  following  method  will  be  useful.  A  mechanism  is  developed 
here  that  enables  the  customer  to  compare  various  contractors’  probabi¬ 
listic  estimates  of  several  attributes  and  characteristics  to  its  own  esti¬ 
mate.  To  use  either  of  the  two  approaches  to  estimate  cost  or  schedule 
variances,  the  contractor  should  familiarize  the  team  making  such  esti¬ 
mation  with  the  intricacy  of  these  approaches  and  alert  it  to  the  cogni¬ 
tive  biases  inherent  in  such  an  estimation  process.  In  their  quest  to 
quantify  these  human  biases,  Alpert  &  Raiffa  (1982)  conducted  several 
experiments  over  two  decades  ago  and  arrived  at  the  conclusion  that 
with  appropriate  training,  the  use  of  the  fractile  method  can  be  very 
effective.  The  question  of  gaming  and  the  manipulation  of  the  approach 
by  some  contractors  to  gain  advantage  is  discussed  under  the  subhead 
“Comparative  Analysis  Among  Contractors." 

The  proposed  methodological  framework  requires  a  number  of 
steps.  First,  the  customer  requires  that  each  bidding  contractor  sub¬ 
mit  either  basic  information  on  the  cost  and  completion  time  vari- 
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ances  or  the  corresponding  PDFs.  In  the  latter  case,  the  contractor 
may  choose  to  use  a  triangular  distribution,  the  fractile  method,  or 
any  other  means  that  the  contractor  believes  will  provide  the  most 
accurate  estimate.  Clearly,  the  contractor  can  and  is  likely  to  use 
models  and  other  tools  to  generate  the  required  PDFs.  At  the  same 
time  the  customer’s  staff  will  generate  its  own  PDFs  for  cost  and 
completion  time.  The  customer  is  now  able  to  compare  not  only  the 
expected  values  (the  means)  of  each  contractor’s  cost  and  comple¬ 
tion  time,  but  also  the  variances  of  these  estimates.  Furthermore,  a 
comparison  of  the  extremes  of  each  PDF  provides  valuable  informa¬ 
tion  to  the  customer  about  each  contractor’s  capabilities  and  compat¬ 
ibility.  Although  some  of  the  efficacious  attributes  of  the  method¬ 
ological  framework  will  be  better  understood  after  we  introduce  the 
section  on  “Risk  of  Extreme  Events”  and  the  example  problems,  an 
overview  of  the  required  steps  may  clarify  the  process: 

1.  Use  the  fractile  method,  the  triangular  distribution,  or  any  other 
approach  to  quantify  the  variances  associated  with  project  cost 
estimates  and  the  completion  schedule. 

2.  Assess  the  contractor’s  capability  to  deliver  the  product  and  to 
estimate  the  likely  variance  of  the  project’s  cost  and  schedule 
through  SEI’s  Software  Risk  Taxonomy-Based  Questionnaire.  The 
SEI  taxonomy  and  the  accompanying  questionnaire  provide  a 
framework  for  identifying  the  technical  uncertainties  in  a  project 
and  the  root  causes  of  these  uncertainties.  It  also  provides  a  method 
for  assessing  the  hone.sty  and  credibility  of  the  contractor’s  analy¬ 
sis  and  figures. 

3.  Evaluate,  in  a  quantitative  way,  any  discrepancy  between  the  variance 
assessments  of  the  contractor  and  the  customer.  (The  quantification 
is  likely  to  lead  to  significant  information  about  the  likelihood  of 
extreme  events  and  their  potential  consequences  for  the  entire  project.) 

4.  Investigate  and  understand  the  contractor’s  assumptions  in  estimating 
variances.  This  information  will  enable  the  customer’s  staff  to  take 
appropriate  measures  to  mitigate  software  nontechnical  risk. 

5.  Integrate  the  information  on  the  contractor  derived  from  (a)  the 
quantitative  variances  received  on  the  projected  cost  and  comple¬ 
tion  schedule  with  (b)  the  results  generated  from  the  Software 
Risk  Taxonomy-Based  Questionnaire. 
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6.  Use  the  risk  ranking  and  filtering  (RRF)  method  to  rank  the  risks 
associated  with  each  prospective  contracting  organization;  then, 
compare  those  risks  against  one  another  and  against  an  estab¬ 
lished  norm. 

Although  these  methods  and  processes  may  not  provide  an  optimal 
approach  for  selecting  the  best  or  most  valid  estimate,  they  do  provide  a 
foundation  that  is  systematic  and  repeatable  to  allow  the  evaluators  to 
gain  significant  knowledge  and  insight  into  the  estimators’  assumptions. 
This  insight  is  critical  from  two  perspectives;  it  enables  the  customer  to: 

•  evaluate  whether  the  contractors'  estimates  and  assumptions  are 
valid  and  consistent  with  the  specifications;  and 

•  establish  a  foundation  by  which  to  evaluate  and  judge  future  changes 
to  cost  and  schedule  estimates. 

These  methods  and  processes  also  provide  a  mechanism  for  the 
customer’s  evaluation  team  to  document  the  assumptions  and  risks  in  a 
cost  or  schedule  estimate,  identify  the  root  issues  associated  with  these 
assumptions  and  risks,  and  organize  this  information  within  the  tax¬ 
onomy  framework.  This  information  can  then  be  used  to  measure 
progress  and  can  also  be  used  as  a  metric  against  future  cost  and  sched¬ 
ule  estimates. 

RISK  OF  EXTREME  EVENTS 

In  general,  the  estimates  of  the  most  likely  project  cost  provided  by  the 
dominant  number  of  contractors  will  be  within  a  close  range  of  one 
another.  Since  assessing  and  ultimately  preventing  potential  major  cost 
overruns  and  time  delays  are  of  major  concern  in  this  paper,  our  interest 
here  is  in  what  can  go  wrong  in  the  extremes,  i.e..  in  the  behavior  of  the 
tail  of  the  distribution.  This  can  best  be  captured  through  the  condi¬ 
tional  expected  value  of  extreme  events.  The  conditional  expected  value 
of  risk,  denoted  by  f4(*)  (which  will  be  defined  later),  can  provide  valu¬ 
able  information  that  supplements  and  complements  the  average  cost  or 
most  likely  cost  estimates. 

Most  analysts,  who  use  probabilistic  quantitative  methods  to  measure 
the  risk  of  project  cost  overruns  and  delays  in  the  completion  schedule, 
resort  to  the  most  common  mathematical  construct  for  the  quantifica¬ 
tion  of  risk — the  expected  value  of  risk.  Whether  the  probabilities  asso¬ 
ciated  with  the  universe  of  events  are  viewed  by  the  analyst  as  discrete 
or  continuous,  the  expected  value  of  risk  is  an  operation  that  essentially 
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multiplies  each  event  by  its  probability  of  occurrence  and  sums  (or  inte¬ 
grates)  all  these  products  over  the  entire  universe  of  events.  This  opera¬ 
tion  literally  commensurates  adverse  events  of  high  consequences  and 
low  probabilities  of  exceedance  with  events  of  low  consequences  and 
high  probabilities  of  exceedance.  Indeed,  the  expected  value  masks  the 
extremes  and  hides  the  effects  of  less  likely  outcomes. 

The  misuse,  misinterpretation,  and  fallacy  of  the  expected  value,  when 
it  is  used  as  the  sole  criterion  for  risk  in  decision  making,  are  discussed 
elsewhere  (Haimes  &  Chittister  1993,  Asbeck  &  Haimes  1984).  Many 
experts  are  becoming  convinced  of  the  grave  limitations  of  the  tradi¬ 
tional  and  commonly  used  expected-value  concept  and  so  arc  augment¬ 
ing  the  expected  value  of  risk  with  a  supplementary  measure — the  con¬ 
ditional  expectation — by  which  decisions  about  extreme  and  catastrophic 
events  arc  not  averaged  out  with  more  commonly  occurring  high-fre- 
qucncy/low-consequence  events. 

The  partitioned  multiobjective  risk  method  (PMRM)  is  a  risk  analysis 
method  developed  for  solving  probabilistic  multiobjective  problems  with 
a  focus  on  extreme  events  (Asbeck  &  Haimes.  1984).  Instead  of  using 
the  traditional  expected  value  of  risk,  the  PMRM  generates  a  number  of 
conditional  expected-value  functions,  known  as  risk  functions,  which  rep¬ 
resent  the  risk  given  that  the  damage  falls  witin  specific  ranges  of  the 
probability  of  exceedance  or  within  a  range  of  adverse  consequences 
(gencrically  termed  as  damages).  Before  the  PMRM  was  developed, 
problems  with  at  least  one  random  variable  were  solved  by  computing 
and  minimizing  the  unconditional  expectation  of  the  random  variable 
repre.senting  the  .specific  damage.  In  contrast,  the  PMRM  isolates  a 
number  of  damage  ranges  (by  specifying  partitioning  probabilities)  and 
generates  conditional  expectations  of  damage  given  that  the  damage 
falls  within  a  particular  range.  In  this  manner,  the  PMRM  can  generate 
a  number  of  risk  functions,  one  for  each  range,  which  are  then  aug¬ 
mented  with  the  original  optimization  problem  as  new  objective  func¬ 
tions.  In  this  paper  the  di.scussion  will  be  limited  to  one  conditional 
expected  value  of  extreme  events,  denoted  by  f4(»). 

The  conditional  expectations  of  a  problem  are  found  by  partitioning 
the  problem  s  probability  axis  and  mapping  these  partitions  onto  the 
damage  axis.  The  damage  axis  in  this  ease  can  be  project  cost  overrun  in 
terms  of  dollars  or  percentage  of  overage  above  the  contracted  level: 
alternatively,  the  damage  can  represent  time  delay  in  terms  of  months 
or  weeks  or  in  terms  of  percentages  in  relation  to  the  original  time 
schedule.  Consequently,  the  damage  axis  is  partitioned  into  correspond¬ 
ing  ranges.  .4  conditional  expectation  is  defined  as  the  expected  value  of  a 
random  variable  t’iven  that  this  value  lies  within  some  prespecified  proh- 
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ability  range  (or  within  some  prespecified  damage  range).  Clearly,  the  val¬ 
ues  of  conditional  expectations  are  dependent  on  where  the  probability 
axis  (or  the  damage  axis)  is  partitioned.  The  choice  of  where  to  partition 
is  made  subjectively  by  the  analyst  in  response  to  the  extreme  character¬ 
istics  of  the  decision-making  problem. 

A  continuous  random  variable  X  of  damages  (e.g.,  cost  overrun  or 
time  delay)  has  a  cumulative  distribution  function  (CDF)  P(x),  and  a 
probability  density  function  (PDF)  p(x),  which  are  defined  by  the  rela¬ 
tionships 


CDF:  P(x)  =  prob[X  <  x] 


PDF;  p(x)  = 


dP  (x) 

dx 


and 


(1) 

(2) 


The  CDF  represents  the  nonexceedance  probability  of  x.  The 
exceedance  probability  of  x  is  defined  as  the  probability  that  X  is  ob¬ 
served  to  be  greater  than  x  and  is  equal  to  one  minus  the  CDF  evaluated 
at  X. 

The  expected  value,  average,  or  mean  value  of  the  random  variable  X 
is  defined  as 


E[X]  =  |p(x)dx 
0 


(3) 


For  the  purpose  of  this  article,  a  modified  version  of  the  PMRM  is 
presented  to  simplity  the  mathematical  discussion  and  to  focus  the  analysis 
on  the  conditional  risk  of  extreme  events.  Let  1  -  a,  where  0  <  a  <  1, 
denote  an  exceedance  probability  that  partitions  the  domain  of  X  into 
two  ranges.  On  a  plot  of  exceedance  probability,  there  is  a  unique  dam¬ 
age  b  on  the  damage  axis  that  corresponds  to  the  exceedance  probability 
1  -  a  on  the  probability  axis.  Damages  (e.g.,  cost  overruns  or  time  delays 
in  project  completion  schedule)  less  than  b  are  considered  to  be  of  low 
to  moderate  severity;  damages  greater  than  b  are  of  high  severity.  The 
partitioning  of  risk  into  two  severity  ranges  is  illustrated  in  Figure  2.  If 
the  partitioning  probability  a  is  specified,  for  example,  to  be  0.95,  then  b 
is  the  95th  percentile. 

The  conditional  expected  damage  (given  that  the  damage  is  within 
that  particular  range)  provides  a  measure  of  the  risk  associated  with  the 
range.  The  measure  of  conditional  expected  value  of  risk  of  interest 
here  is  that  of  low  exceedance  probability  and  high  severity,  denoted  by 
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f4(*).  High  severity  may  mean  a  high  cost  overrun  or  a  high  time  delay  in 
the  project’s  scheduled  completion.  The  function  f4(»)  is  the  expected 
value  of  X,  given  that  x  is  greater  than  or  equal  to  b: 

f4(.)  =  E[X  I  X  >  b]  (4) 


For  any  probability  of  exceedance,  one  can  generate  the  traditional, 
unconditional  (common)  expected  value  of  risk  (of  cost  overrun  and/or 
of  time  delay)  denoted  by  f5(«),  and  the  conditional  expected  value  of 
risk  of  extreme  events  (of  same)  denoted  by  f4(»).  Note  that 


f4(.)  = 


(5) 


f5(«)  =  jp(x)dx 
0 


where  p(x)  is  the  probability  density  function. 


EXAMPLE  PROBLEM 

DoD  is  considering  the  introduction  of  a  new  strategic  airplane  that  will 
constitute  the  flagship  of  the  Air  Force  as  we  enter  the  third  millen¬ 
nium.  Aware  of  the  powershift  from  hardware  to  software  in  technology 
and  the  emerging  centrality  of  software  as  the  overall  system  integrator 
and  coordinator,  DoD  considers  the  development  of  software  for  this 
airplane  to  be  of  paramount  importance  (Chittister  &  Haimes  1994). 
The  Air  Force  commissions  the  assistance  of  a  support  organization  to 
develop,  in  collaboration  with  its  own  staff,  specifications  and  a  request 
for  proposal  (RFP)  for  designing,  prototyping,  and  developing  the  soft¬ 
ware  needed  for  the  flagship  airplane.  Following  a  detailed  and  tedious 
process  of  qualifying  prospective  bidders,  the  Air  Force  issues  an  RFP 
for  the  development  of  the  required  software  engineering.  This  time, 
however,  the  RFP  includes  items  that  had  not  been  requested  previ¬ 
ously.  For  example,  the  RFP  requires  that  each  contractor  provide  vari¬ 
ances  along  with  the  estimated  project’s  cost  and  completion  schedule, 
instead  of  the  commonly-practiced  requirement  of  single  deterministic 
values.  The  RFP  leaves  it  up  to  the  contractors  to  determine  the  form 
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that  these  variances  take,  including,  if  the  contractor  so  desires,  the  type 
of  PDF  selected  for  each  estimate.  The  Air  Force  and  its  support  team, 
planning  to  use  the  same  approach  themselves  in  evaluating  the  various 
proposals,  recommends  in  the  RFP  the  optional  use  of  the  fractile 
method  or  the  triangular  distribution  when  complete  statistical  informa¬ 
tion  is  not  readily  available. 

To  capture  the  mathematical  details  entailed  in  the  process  of  devel¬ 
oping  representative  probability  density  functions  (PDFs)  for  cost  and 
completion  time,  a  step-by-step  procedure  using  the  fractile  method 
(adopted  by  Contractors  A  and  B)  is  presented  here.  For  pedagogical 
purposes,  a  detailed  analysis  is  presented  for  Contractor  A  only.  Similar 
analysis  should  be  followed  for  Contractor  B  and  the  customer.  The 
team  from  Contractor  A  estimates  a  most  likely  cost  of  $150  million.  After 
considerable  brainstorming  sessions,  the  following  information  emerges: 

•  Best  case  project  cost  increase  =  0%  (i.e.,  project  cost  is  $150 
million); 

•  Worst  case  project  cost  increase  =  50%  (i.e.,  project  cost  increase 
is  $75  million,  for  a  total  of  $225  million);; 

•  Median  value  of  project  cost  increase  (equal  likelihood  of  being 
greater  or  less  than  this  value)  =  15%  (i.e.,  project  cost  increase  is 
$22.5  million,  for  a  total  of  $172.5  million); 

•  50-50  chance  that  the  actual  project  cost  would  be  within  5%  of  the 
15%  median  estimate  (i.e.,  project  cost  increase  is  (15  ±  5)%). 

From  the  above  information,  the  following  fractiles  (percentiles)  are 
readily  determined. 

•  The  best  scenario  of  no  cost  overrun  (0%  cost  increase,  i.e.,  a  total 
cost  of  $150  million)  represents  the  0.00  fractile  (0  percentile); 

•  The  worst  scenario  of  50%  cost  overrun  (a  total  cost  of  $225  mil¬ 
lion)  represents  the  1.00  fractile  (100th  percentile); 

•  The  median  value  of  15%  cost  overrun  (a  total  cost  of  $172.5  mil¬ 
lion)  represents  the  0.50  fractile  (50th  percentile); 

•  The  0.25  fractile  (25th  percentile)  is  (15-5)%  =  10%  increase  over 
$150  million  (a  total  cost  of  $165  million). 
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•  The  0.75  fractile  (75th  percentile)  is  (15+5)%  =  20%  increase  over 
$150  million  (a  total  cost  of  $180  million). 

The  above  assessment  of  project  cost  for  contractor  A  and  similarly 
for  Contractor  B  and  for  the  customer  is  summarized  in  Table  1  and  is 
used  as  a  basis  for  constructing  the  corresponding  cumulative  distribu¬ 
tion  function  (CDF)  for  Contractor  A.  (See  Figure  3.) 

The  CDF  (Figure  3)  can  now  be  represented  in  terms  of  a  PDF 
(Figure  4).  To  construct  the  PDF,  one  must  be  guided  by  the  following 
principles: 

(a)  The  area  under  the  shaded  area  (the  PDF)  must  be  equal  to  1. 

(b)  The  first  quartile  in  Figure  3  (representing  25%  of  the  probabili¬ 
ties)  spans  a  cost  overrun  from  0%  to  10%.  Thus,  the  correspond¬ 
ing  area  of  the  PDF  (Figure  4)  must  be  equal  to  one-fourth  of  the 
total  area,  i.e.,  0.25.  Dividing  0.25  by  10  yields  a  height  of  0.025  for 
the  first  rectangle  in  Figure  4.  Similarly  for  the  other  three  quartiles, 
each  of  the  second  and  third  quartiles  spans  5%  of  project  cost 
increase.  Thus,  the  height  of  the  rectangle  of  the  PDF  (Figure  4) 
is  0.25  on  the  probability  axis  and  when  divided  by  5  yields  a 
height  of  0.05  on  the  probability  axis.  Finally,  the  last  quartile 
spans  a  cost  overrun  of  30%  (from  20%  to  50%).  Thus,  the 
height  of  the  rectangle  (on  the  probability  axis)  is  0.25  divided 
by  30  which  yields  a  height  of  0.008.  Figure  5  depicts  the 
exceedance  probability  (1-CDF)  vs.  project  cost  increase.  To  fo¬ 
cus  on  the  exceedance  probability  of  a  major  cost  overrun,  say 
between  20%  and  50%,  only  that  part  of  Figure  5  is  depicted  in 
Figure  6.  Note  that  by  just  using  basic  rules  from  geometry,  one 
can  relate  the  exceedance  probability  to  any  project  cost  increase 
X,  for  20%  _  X  _  50%. 

The  expected  value  of  the  percentage  of  project  cost  increase,  f5(«), 
can  be  determined  from  geometry  (Figure  4); 

(10-0)  (15-10)  (20-15) 

f5(*)  =  0.25  [  0  +  — y—  ]  +  0.25  [10  +  +  0.25  [15  +  — y— 

(50  -  20) 

+  0.25  [20  -f  — 2 ] 

=  0.25  (5)  +  0.25  (12.5)  +  0.25  (17.5)  +  0.25  (35) 

=  0.25  (70)  =  17.5%  (i.e.,  total  cost  of  $(150  +  26.25)  million) 
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The  expected  value  of  the  percentage  of  project  cost  increase  may 
also  be  calculated  using  Equation  3.  Note  that  the  expected  value  of  cost 
overrun  of  $26.25  million  (i.e.,  total  cost  of  $176.25  million)  for  Contrac¬ 
tor  A  does  not  provide  any  vital  information  on  the  probable  extreme 
behavior  of  project  cost.  Also  note  that  there  is  a  one-to-one  functional 
relationship  between  the  probability  axis  and  the  percentage  of  project 
cost  increase  as  is  depicted  in  Figure  5.  For  example,  there  is  0.1  prob¬ 
ability  (one  chance  in  10)  that  project  cost  increase  will  be  equal  to  or 
above  38%.  This  result  is  generated  as  follows  (here  we  are  interested  in 
the  probability  of  exceedance  of  0.1,  i.e.,  a  =  0.90,  or  (1-a)  =  0.10): 

x-20  ^  ah  ^  0.25  -(1  -a) 

50  -  20  ac  0.25 

30(1 -a) 

Thus,  X  =  30  -  0  25  +  20  =  38%  for  a  =  0.9 

Alternatively,  we  can  compute  from  Figure  6  the  partition  point  x 
(the  percentage  of  increase  in  cost)  that  corresponds  to  a  probability  of 
0.1  as  shown  below: 

(1  -  a)  =  (50  -  x)  h 

where  h  is  the  height  in  the  probability  axis 
0.25 

^  ~  50  -  20  ■  ^^-0083 

X  =  50  -  ( -^)  =  50  -  =  38%  for  a  =  0.9 

The  conditional  expected  value  of  project  cost  can  be  calculated  for  a 
couple  of  scenarios  to  shed  light  on  the  behavior  of  the  tail  of  the  PDF. 
For  example,  given  (from  Figures  5  and  6)  that  there  is  0.1  probability 
of  project  cost  overrun  that  would  be  equal  to  or  exceed  38%  of  its 
original  scheduled  budget,  management  might  be  interested  in  answer¬ 
ing  the  following  question:  What  is  the  conditional  expected  value  of 
extreme  cost  overrun  beyond  the  38%  (or  extreme  cost  overrun  with 
exceedance  probability  that  is  below  0.1)?  Or  posed  differently,  within 
the  range  of  exceedance  probabilities  between  0.1  and  0.0  and  range  of 
cost  overruns  between  38%  and  50%,  what  is  the  expected  value  of 
project  cost  overrun?  Note  that  (a)  the  maximum  cost  overrun  was  pre¬ 
dicted  not  to  exceed  5()9f ;  (b)  the  conditional  expected  value  is  the 
eommon  expected  value  limited  between  specific  levels  of  cost  overruns 
instead  of  the  entire  range  of  possible  cost  overruns;  and  (c)  the  ex- 
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pectcd  value  is  a  weighted  average  of  possible  cost  overruns  multiplied 
by  their  corresponding  probabilities  of  occurence  and  summed  over  that 
entire  range. 

Using  Equation  3,  the  common,  unconditional  expected  value  of  cost 
overrun,  f5(»),  was  calculated  earlier  to  be  17.5%.  Similarly,  the  condi¬ 
tional  expected  value  of  cost  overrun  under  the  scenario  of  0.1  probabil¬ 
ity  of  exceeding  the  original  cost  estimate  (by  38%  or  by  $57  million) 
computed  using  equation  5  yields  f4(»)  =  44%.  Note  that  the  PDF  of 
cost  overrun  portion  from  20%  and  beyond  is  a  linear  function.  Alterna¬ 
tively,  the  conditional  expected  value  can  be  computed  as  the  mean  of 
the  shaded  area  in  Figure  7. 


f4(*)  =  38  + 


(50^)^  44% 


In  other  words,  the  adjusted  (conditional)  expected  value  of  cost  over¬ 
run.  when  it  is  in  the  range  of  38%  to  50%  of  the  original  scheduled 
cost,  is  44%. 

Unless  the  project  is  a  cost-plus  contract,  the  interpretation  of  these 
results  should  alarm  top  management  of  Contractor  A;  although  the 
expected  cost  overrun  of  the  proposed  budget  is  17.50%  above  the  bud¬ 
geted  cost  of  $150  million,  there  is  a  10%  chance  (0.1  probability)  that 
the  cost  overrun  will  exceed  38%  of  the  budgeted  cost!  Furthermore,  at 


Project  Cost  increase  (%) 


Fii^urc  5.  Exceedance  Probability  For  Project  Cost  Increase 

for  Contractor  A 
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Figure  6.  Computing  the  Partition  Point  on  the  Damage  Axis 

for  Contractor  A 


Figure  7.  Computing  the  Conditional  Expected  Value  for  Contractor  A 
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a  10%  chance  of  cost  overrun,  the  conditional  expected  value  of  cost 
overrun  that  exceeds  38%  is  44%  above  the  original  budget,  i.e.,  an 
exceedance  of  $66  million;  in  other  words,  under  these  conditions,  the 
conditional  expected  value  of  the  total  cost  will  be  ($150  +  66)  million 
=  $216  million. 

It  is  worthwhile  to  clarify  at  this  point  the  meaning  of  the  two  distinct 
terms  of  cost  overrun:  38%  and  44%.  The  term  38%  cost  overrun  corre¬ 
sponds  to  a  single  probability  point  and  is  derived  directly  from  Figure  6. 
The  term  44%,  on  the  other  hand,  represents  the  conditional  expected 
value,  the  averaging  of  all  the  probabilities  from  0.10  to  zero  multiplied 
by  the  corresponding  cost  overruns  from  38%  to  50%,  summed  as  ap¬ 
propriate  and  scaled.  Thus 

f4(«)  =  E  [X  1  >  38%  cost  overrun]  =  44% 


or  equivalently, 

f4(*)  =  E  [X  I  >  $207  million]  =  $66  million 

It  is  constructive  to  further  clarify  the  information  summarized  in 
Table  2.  Consider  the  customer’s  column.  According  to  the  customer’s 
estimates  the  common,  unconditional  expected  value  of  cost  overrun  is 
11.25%.  Through  mathematical  calculations  on  the  basis  of  the  informa¬ 
tion  provided  by  the  customer  (as  shown  in  Table  1),  it  can  be  deter¬ 
mined  that  there  is  0.1  probability  of  project  cost  overrun  that  would 
exceed  24%  of  its  original  scheduled  cost  (sec  Haimes  and  Chittister 
1993).  Thus,  the  conditional  expected  value  of  extreme  cost  overrun 
between  24%  and  30%  (or  extreme  cost  overrun  with  exceedance  prob¬ 
ability  that  is  below  0.1)  is  27%. 

COMPARATIVE  ANALYSIS  AMONG  CONTRACTORS 
An  important  activity  within  Phase  III  of  the  four-phase  acquisition 
process  is  understanding  the  reasons  and  the  genesis  for  the  variations 
of  the  estimates  among  the  contractors  and  explaining  these  differences 
on  the  basis  of  the  evidence  collected  through  the  Taxonomy,  the  inter¬ 
views,  and  other  ways. 

1.  The  contractor  knows  what  is  to  be  known  about  the  project. 

2.  The  contractor  does  not  know  what  is  to  be  known  about  the 
project. 
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Table  1. 

COMPARATIVE  TABULAR  CDF 


Fractiie 

Project  Cost  Increase  (%) 

Customer 

Contractor  A 

Contractor  B 

0 

0 

0 

0.25 

5 

10 

15 

0.50 

10 

15 

20 

0.75 

15 

20 

25 

1.00 

30 

50 

40 

3.  The  contractor  knows  and  is  aware  of  the  unknowns  and  uncer¬ 
tainties  about  the  project. 

4,  The  contractor  does  not  know  and  is  not  aware  of  the  uncertain¬ 
ties  surrounding  the  project. 

Of  course  this  knowledge  or  the  lack  thereof  is  not  absolute  and  the 
contractor’s  own  knowledge  may  be  at  various  levels  between  complete 
awareness  and  complete  ignorance.  This  discussion  assumes  that  no  gam¬ 
ing  is  taking  place.  Safeguards  must  be  developed,  however,  to  secure 
against  a  contractor  who  opts  to  game  the  system. 

With  these  and  other  comparisons  that  can  be  made  as  desired,  the 
customer’s  ability  to  assess  the  various  risks  and  thus  to  mitigate  and 
manage  them  is  greatly  enhanced.  For  example,  when  there  appears  to 
be  a  substantial  discrepancy  either  between  the  customer’s  and  one  of 
the  contractor's  estimates  or  among  the  estimates  of  the  various  con¬ 
tractors,  the  customer  can  inquire  (if  legally  permissible)  about  evidence 
and  sources  of  these  variations;  otherwise,  the  customer  can  draw  con¬ 
clusions  on  the  contractor’s  estimation  capabilities  and  honesty.  The  use 
of  SEI’s  software  Risk  Taxonomy-Based  Questionnaire  [SEI  1993]  in 
conjunction  with  these  analyses  will  be  discussed  in  a  subsequent  sec¬ 
tion. 
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Table  2. 

SUMMARY  OF  RESULTS 


Customer 

Contractor  A 

Contractor  B 

Unconditional 
Expected  Value, 
f5(.) 

11.25% 

17.50% 

20% 

Partitioning 

Point 

a  =  0.90 

a  s  0.90 

a  B  0.90 

Corresponding 
Percent  of  Cost 
Increase 

X  =  24% 

X  =:  38% 

X  =  34% 

Conditional 
Expected  Value, 
f4(.) 

27% 

44% 

37% 

The  methodology  advocated  in  this  paper  does  not  embrace  an 
adversarial  relationship  between  the  customer  and  the  prospective  con¬ 
tractors.  Project  cost  overrun  and  schedule  time  delay  are  not  assumed 
to  happen  necessarily  because  of  contractors’  conspiracy  or  malice. 
Rather,  the  premise  here  is  that  often  the  customer  and  the  contractors 
do  not  adhere  to  a  systemic  risk  assessment  approach  in  their  evaluation 
and  projection  of  software  nontechnical  risk,  and  the  unintended  result 
is  a  cost  overrun  or  delay  in  project  completion  schedule. 

The  use  of  the  Software  Risk  Taxonomy-Based  Questionnaire  devel¬ 
oped  by  the  Software  Engineering  Institute  constitutes  the  basis  for  this 
needed  systemic  risk  assessment  approach  (Carr  et  al.  1993).  The  SEI 
taxonomy  questionnaire  is  divided  into  three  major  parts:  Product  Engi¬ 
neering,  Development  Environment,  and  Program  Constraints. 

The  primary  focus  of  the  SETs  risk  identification  process  is  to  elicit 
known  and  unknown  risks  from  the  personnel  associated  with  the  project, 
e.g.,  administrative  and  technical  management,  development  engineers, 
proposal  team,  and  cost  estimators.  The  identification  process  consists 
of  a  taxonomy-based  questionnaire  and  a  method  for  conducting  inter¬ 
views  using  this  questionnaire.  This  enables  the  interviewers  to  probe 
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for  both  technical  and  nontechnical  risks  that  affect  the  project.  The 
information  that  is  gathered  from  the  interviews  can  be  grouped  and 
ordered  using  a  set  of  criteria  and  a  risk  ranking  and  filtering  method. 
The  strength  of  this  approach  is  that  the  process  is  repeatable  and  sys¬ 
tematic,  and  it  enables  the  analysis  and  comparison  of  data  from  mul¬ 
tiple  organizations.  The  analysis  and  comparison  of  risks  and  concerns 
coupled  with  the  extreme  events  information  will  provide  the  customer 
with  a  foundation  upon  which  to  make  more  informed  decisions  regard¬ 
ing  the  risks  in  the  cost  or  schedule  estimates. 

An  additional  benefit  of  the  analysis  is  that  customers  can  gain  valu¬ 
able  insight  into  the  contractors’  assumptions  and  the  depth  of  their 
understanding  regarding  the  requirements  and  technology  associated 
with  the  project. 

CONCLUSION 

Controlling  the  cost  and  time  schedule  of  major  projects  has  been  and 
continues  to  be  a  major  problem  facing  government  and  non-govcrn- 
ment  acquisition  managers.  Software  development  projects  are  no  ex¬ 
ception.  Because  of  the  close  influence  and  interaction  between  soft¬ 
ware  technical  and  nontechnical  risks  and  the  diverse  sources  and  causes 
that  constitute  the  driving  force  behind  these  risks,  the  acquisition 
manager’s  job  is  complicated.  One  of  the  major  premises  of  this  paper  is 
that  a  careful,  systemic,  and  analytically-based  process  for  contractor 
selection  is  imperative  for  the  prevention  of  major  risks  of  cost  overruns 
and  time  delays.  This  paper  proposes  such  an  acquisition  process.  The 
four-phase  process  can  be  best  viewed  as  a  framework  rather  than  as  a 
rigid  step-by-step  procedure.  The  obvious  limitation  in  the  scope  of  any 
single  paper  prevents  a  full  demonstration  of  each  of  the  four  phases  of 
the  proposed  acquisition  process.  The  three  sample  problems  should 
successfully  communicate  to  the  reader  the  mathematical  mechanics  as¬ 
sociated  with  Phase  I  and  the  construction  of  the  measure  of  risk  of 
extreme  events  in  Phase  II.  The  readers  who  are  more  familiar  with  the 
SEI  Taxonomy-Based  Questionnaire  will  be  able  to  relate  more  easily  to 
its  use  in  Phases  II  and  III.  Similar  statements  can  be  made  on  the 
familiarity  with  the  risk  ranking  and  filtering  method,  the  independent 
verification  and  validation  team,  and  with  other  methods  used  in  the 
proposed  acquisition  process.  As  a  framework,  the  discussion  in  this 
paper  must  be  construed  by  the  reader  as  the  beginning  of  a  dialogue 
toward  the  quantification  and  management  of  the  risks  of  cost  overruns 
and  time  delays  associated  with  software  development.  In  this  spirit,  we 
consider  this  paper  to  be  a  precursor  which  will  be  followed  by  others  in 
the  future.  The  expected  benefits  that  result  from  the  prevention  of 
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major  and  extreme  risks,  combined  with  the  low  expected  cost  of  early 
mitigation  strategies,  encourage  us  to  believe  that  this  area  is  worthy  of 
much  further  consideration. 
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rhe  earned  value  method  is  an  internationally  recognized  project 
managment  tool  for  measuring  progress  on  projects.  Despite  its 
popularity,  it  has  not  been  widely  applied  on  software  development 
projects.  This  paper  proposes  the  use  of  earned  value  on  software  develop¬ 
ment  projects.  After  a  brief  description  of  the  earned  value  method,  seven 
software  metrics  appropriate  for  earned  value  application  are  described. 
The  use  of  these  metrics  should  facilitate  more  effective  management  of 
software  development  projects. 


INTRODUCTION 

Measuring  progress  on  software  development  projects  is  a  difficult  but 
important  challenge  for  project  managers.  In  the  Department  of  De- 
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fense  (DoD),  computer  software  costs  arc  a  substantial  and  growing 
portion  of  the  budget.  In  1993,  for  example,  DoD  software  development 
costs  are  estimated  at  over  $30  billion  (Defense  Systems  Management 
College  [DSMC],  1990).  Similar  trends  are  apparent  in  high-tech  com¬ 
mercial  projects. 

Since  1967,  the  DoD  has  used  a  performance  measurement  technique 
known  as  “earned  value”  to  monitor  performance  on  defense  contracts. 
In  the  DoD,  the  earned  value  method  is  usually  implemented  with  Cost/ 
Schedule  Control  Systems  Criteria  (C/SCSC).  However,  the  method  does 
not  require  C/SCSC.  As  a  result,  earned  value  is  rapidly  becoming  an 
internationally  recognized  tool  in  project  management,  with  both  de¬ 
fense  and  nondefense  applications. 

Despite  the  established  utility  of  the  earned  value  method,  its  use  on 
software  development  projects  has  not  been  widespread.  Based  on  dis¬ 
cussions  with  DoD  project  managers  and  analysts,  software  develop¬ 
ment  progress  is  often  assumed  to  be  unmeasurable,  and  software  projects 
are  classified  as  “level-of-effort.”  Given  the  relative  importance  and  cost 
of  these  projects,  arbitrarily  classifying  them  as  “level-of-effort”  is  ex¬ 
tremely  unfortunate. 

This  paper  proposes  the  use  of  the  earned  value  method  to  measure 
progress  on  software  development  projects.  After  a  brief  description  of 
the  earned  value  method  and  related  topics,  seven  software  metrics  are 
described  and  evaluated  for  their  appropriate  application  in  a  perfor¬ 
mance  measurement  system  that  is  based  on  the  earned  value  method. 

BACKGROUND 

Performance  Reporting  and  C/SCSC 

To  facilitate  the  effective  cost  management  of  defense  acquisitions,  the 
DoD  requires  standardized  cost  management  reports  from  defense  con¬ 
tractors.  Two  reports  that  specifically  focus  on  cost  and  schedule  perfor¬ 
mance  are  the  Cost  Performance  Report  (CPR)  and  the  Cost! Schedule 
Status  Report  (C/SSR).  The  CPR  is  normally  submitted  on  contracts 
which  require  compliance  with  the  Department  of  Defense  Cost/Sched- 
ulc  Control  Systems  Criteria  (C/SCSC).  For  contracts  not  required  to 
comply  with  C/SCSC,  the  C/SSR  is  usually  required. 

Compliance  with  the  C/SCSC  is  required  on  “significant”  contracts 
and  subcontracts  within  all  acquisition  programs,  including  those  that 
require  software  development.  DoD  Instruction  5000.2  defines  signifi¬ 
cant  contracts  as  research,  development,  test,  and  evaluation  contracts 
with  an  estimated  cost  of  $60  million  or  more  (in  fiscal  year  1990  con¬ 
stant  dollars),  or  procurement  contracts  with  an  estimated  cost  of  $250 
million  or  more  (in  tlscal  year  1990  constant  dollars).  For  contracts 
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below  these  thresholds,  compliance  with  C/SCSC  may  also  be  required 
when  contract  risk  is  judged  to  be  high.  Compliance  with  C/SCSC  on 
firm  fixed  price  contracts  is  not  normally  required  (Department  of  De¬ 
fense  [DoD],  1991,  February). 

Cost/Schedule  Control  Systems  Criteria  are  not  a  management  sys¬ 
tem  imposed  by  the  government  on  the  contractor.  Instead,  the  criteria 
establish  minimal  standards  for  the  contractor’s  existing  planning,  sched¬ 
uling,  budgeting,  accounting,  and  analysis  systems,  collectively  termed 
the  contractor’s  “internal  management  control  systems.”  In  total,  there 
are  35  rather  generic  standards.  One  criterion,  for  example,  requires  a 
comprehensive  budget  for  all  the  authorized  work  on  the  contract.  An¬ 
other  criterion  requires  that  all  the  authorized  work  be  scheduled. 

The  DoD  specifies  two  objectives  for  the  criteria:  (a)  for  contractors 
to  use  effective  internal  cost  and  schedule  management  control  systems; 
and  (b)  for  the  government  to  be  able  to  rely  on  timely  and  verifiable 
data  produced  by  those  systems  for  determining  product-oriented  con¬ 
tract  status  (Department  of  the  Air  Force  [DAF],  1989,  October).  Im¬ 
plicit  in  these  objectives  is  the  assumption  that  if  the  contractor’s  man¬ 
agement  control  systems  comply  with  the  criteria,  then  the  data  gener¬ 
ated  by  those  systems  are  reliable. 

The  cost  management  report  summarizes  the  contract’s  cost  and  sched¬ 
ule  performance  using  the  key  data  elements  shown  in  Figure  1.  The 
Budgeted  Cost  of  Work  Scheduled  (BCWS)  is  the  sum  of  budgets  allo¬ 
cated  to  time-phased  elements  of  work  on  the  contract,  known  as  work 
packages.  The  cumulative  expression  of  these  budgets,  termed  the  “Per¬ 
formance  Measurement  Baseline,”  takes  on  a  characteristic  S-shaped 
cun'c.  The  end  point  of  the  baseline,  termed  the  “Budget  at  Comple¬ 
tion”  (BAC),  represents  the  total  budget  of  all  the  identified  work  on 
the  contract. 

Another  key  data  element  is  the  Budgeted  Cost  of  Work  Performed 
(BCWP).  BCWP,  also  known  as  “earned  value,”  is  the  same  number  as 
BCWS.  They  are  both  the  budgeted  cost  of  work.  The  only  difference  is 
when  they  are  recorded.  BCWS  is  recorded  when  work  is  planned  to  be 
completed;  BCWP  is  recorded  when  work  is  actually  completed.  If  work 
is  completed  at  a  different  time  from  when  it  was  planned  to  be  com¬ 
pleted,  then  a  “schedule  variance”  is  identified.  Figure  1,  for  example, 
illustrates  an  adverse  schedule  variance  because  cumulative  BCWS  ex¬ 
ceeds  cumulative  BCWP.  When  all  of  the  work  on  the  contract  is  com¬ 
pleted.  cumulative  BCWP  will  equal  cumulative  BCWS. 

Figure  1  illustrates  two  other  variances:  cost  variance  and  variance  at 
completion.  A  cost  variance  is  the  difference  between  BCWP  and  Ac¬ 
tual  Cost  of  Work  Performed  (ACWP).  In  this  example,  the  cost  vari- 
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Figure  1.  The  Performance  Measurement  Baseline  (PMB) 


ance  is  unfavorable  because  the  actual  cost  exceeds  the  budgeted  cost  of 
the  completed  work.  The  variance  at  completion  is  the  difference  be¬ 
tween  the  Estimate  at  Completion  (EAC)  and  the  Budget  At  Comple¬ 
tion  (BAC). 

The  EAC  is  simply  the  actual  cost  of  completed  work  plus  an  estimate 
of  the  cost  to  complete  the  remaining  work  on  the  contract.  This  esti¬ 
mate  is  reported  by  the  contractor  on  the  cost  management  report  and 
reviewed  for  reasonableness  by  the  government.  When  this  projected 
final  cost  exceeds  the  budget,  the  contractor  is  effectively  predicting  an 
overrun,  termed  an  adverse  “variance  at  completion.”  Figure  1  illus¬ 
trates  the  usual  condition  of  a  defense  acquisition  contract:  behind  sched¬ 
ule  and  overrunning  the  budget  (Christensen,  Antolini,  and  McKinney, 
1992). 

The  C/Sese  require  that  all  “significant”  variances  on  the  contract 
be  analyzed.  By  definition,  a  significant  variance  is  one  that  breaches  a 
threshold  (DAF,  1987,  October,  pps.  3-17).  Thresholds  arc  usually  ex¬ 
pressed  as  a  percentage  and  in  dollars.  If,  for  example,  a  threshold  for  a 
work  package  was  ±10  percent  and  $10,000  dollars,  then  any  variance 
that  breached  thiss  threshold  would  be  investigated  and  it  is  to  be  hoped, 
corrected.  The  intent  is  that  though  dhsciplincd  variance  analysis,  prob- 
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lems  can  be  corrected  before  they  become  serious. 

Clearly,  for  variance  analysis  to  be  effective,  the  proper  planning  and 
measurement  of  earned  value  is  essential.  One  of  the  purposes  of  the 
criteria  (C/SCSC)  is  to  assure  that  the  earned  value  method  is  properly 
planned  and  implemented.  Earned  value  (BCWP)  is  the  key  number  on 
the  cost  management  report.  If  it  is  inaccurate,  then  the  three  variances 
and  the  EAC  are  also  wrong.  It  is  possible,  however,  to  use  the  earned 
value  method  without  the  criteria.  When  this  is  the  case,  controls  similar 
to  those  described  by  the  criteria  should  be  enforced.  Otherwise,  BCWP 
will  not  be  a  reliable  indicator  of  progress  on  the  project.  This  paper  will 
now  describe  how  BCWP  is  planned  and  measured. 

Planning  and  Measuring  Earned  Value 

As  described  earlier,  the  criteria  require  that  all  the  work  on  a  contract 
be  budgeted  and  scheduled.  To  accomplish  this,  the  contractor  will  first 
develop  a  product-oriented  family  tree  of  hardware,  software  and  ser¬ 
vices  that  successively  subdivides  all  of  the  authorized  work  on  the  con¬ 
tract.  This  detailed  breakdown  of  the  work,  termed  the  “Contract  Work 
Breakdown  Structure”  (CWBS),  typically  extends  to  levels  where  work 
is  to  be  performed,  called  “work  packages.”  There  may  be  over  100,000 
work  packages  on  a  large  defense  acquisition  contract. 

A  work  package  has  three  characteristics:  technical  content,  schedule, 
and  budget.  Once  the  contract  is  subdivided  into  work  packages,  each 
work  package  is  arranged  in  the  order  that  it  has  to  be  accomplished, 
assigned  start  and  stop  dates,  and  assigned  a  budget.  The  budget  for 
each  work  package  is  then  spread  through  the  life  of  the  work  package 
according  to  the  technical  requirements  of  the  work.  These  “time-phased” 
budgets  for  all  work  packages  become  the  basis  for  monthly  BCWS, 
monthly  BCWP,  and  the  Performance  Measurement  Baseline.  The  proper 
time-phasing  of  the  budget  is  thus  critical  to  the  planning  of  BCWS  and 
the  subsequent  measurement  of  BCWP. 

There  are  many  “earned  value  methods”  to  time-phase  the  budget  for 
BCWS  and  BCWP  (Fleming,  1992,  pps.  119-127),  As  indicated  in  Table  1, 
earned  value  methods  depend  upon  the  nature  of  the  work  that  is  being 
measured.  Progress  on  the  contract  should  ideally  be  measured  by  as¬ 
sessing  discrete  tasks  which  have  a  specific  end  product  or  end  result. 
Work  of  this  kind  is  termed  “discrete  effort.”  Common  earned  value 
methods  appropriate  for  discrete  effort  include  weighted  milestones, 
interim  milestones,  and  percent  complete. 

Work  that  can  be  directly  related  to  other  identified  discrete  tasks, 
such  as  quality  control  or  inspection,  is  termed  “apportioned  effort.” 
Support  type  activities,  such  as  sustaining  engineering  or  coordination. 
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TABLE  I 

EARNED  VALUE  METHODS 


Category  of  Work 
Discrete  Effort 


Apportioned  Effort 
Level  of  Effort 


Earned  Value  Method 
Weighted  Milestones  (e.g.,  50-50) 
Interim  Milestones 
Percent  Complete 
“Factored”  on  Discrete  Effort 
BCWP  set  equal  to  BCWS 


that  does  not  result  in  a  final  end  product  is  termed  “level  of  effort” 
(LOE).  On  criteria-compliant  contracts,  these  categories  are  mutually 
exclusive  and  collectively  exhaustive.  All  work  must  be  classified  into 
one  of  these  categories. 

Although  the  criteria  allow  the  contractor  to  use  any  one  or  any 
combination  of  these  earned  value  methods,  there  are  some  general 
requirements.  These  requirements  are  intended  to  insure  the  usefulness 
of  the  performance  measurement  data. 

To  be  useful  for  performance  measurement,  the  data  must  be  verifi¬ 
able  and  objective.  Therefore,  the  contractor  must  document  the  earned 
value  method  used  in  developing  BCWS  before  the  work  begins,  and 
then  use  the  same  method  for  measuring  BCWP  when  work  is  being 
performed.  Because  BCWS  and  BCWP  are  the  same  number,  it’s  abso¬ 
lutely  essential  that  the  same  method  be  used  for  each.  In  addition, 
allowing  one  method  for  BCWS  and  another  for  BCWP  would  allow  the 
contractor  to  distort  performance  measurement  and  the  variances  re¬ 
ported  on  the  cost  management  report. 

In  addition  to  being  verifiable  and  objective,  the  numbers  for  BCWP 
must  be  valid;  namely,  BCWP  must  clearly  reflect  performance.  There¬ 
fore,  the  use  of  arbitrary  measurement  methods,  such  as  the  weighted 
milestone  method,  are  limited  to  short-span  work  packages.  An  example 
of  an  arbitrary  weighted  milestone  method  is  the  “50-50”  method,  where 
one  half  of  the  budget  for  the  work  is  “earned”  (recorded  as  BCWP) 
when  the  work  begins,  and  the  other  half  is  earned  when  the  work  is 
completed.  To  minimize  the  dkstortion  created  by  such  an  arbitrary'  per¬ 
formance  measurement,  the  method  is  generally  restricted  to  work  pack¬ 
ages  with  durations  of  two  months  or  le.ss. 

For  longer  work  packages,  “interim  milestones”  are  required,  where  a 
portion  of  the  budget  for  the  work  is  assigned  to  each  milestone.  When 
that  milestone  is  accomplished,  the  budget  for  that  milestone  is  recorded 
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as  BCWP.  As  long  as  the  milestones  are  tangible  and  integral  parts  of 
the  work,  this  interim  milestone  method  will  properly  reflect  perfor¬ 
mance  on  long-span  work  packages. 

For  some  work  packages,  identifying  interim  milestones  may  not  be 
possible.  In  this  case,  the  contractor  may  simply  estimate  the  percentage 
of  the  work  planned  to  be  completed  for  planning  BCWS,  and  later 
estimate  the  percentage  of  work  actually  completed  for  recording  BCWP. 
It  is  to  be  hoped  that  the  contractor  will  employ  some  objective  param^ 
eter  of  progress  as  a  basis  for  estimating  the  percent  complete.  In  any 
case,  the  criteria  require  that  the  contractor’s  method  for  determining 
BCWP  be  rational.  The  contractor  should,  therefore,  be  able  to  explain 
the  basis  for  determining  the  estimates  of  BCWS  and  BCWP. 

Another  requirement  related  to  earned  value  methods  involves  the 
proper  matching  of  ACWP  with  BCWP.  To  facilitate  the  timely  analysis 
of  cost  variances,  ACWP  should  be  recorded  in  the  same  period  that 
BCWP  is  recorded.  When  BCWP  for  a  work  package  is  recorded  but 
the  actual  cost  is  not  yet  known,  ACWP  may  be  estimated.  Later,  when 
the  actual  cost  is  known,  ACWP  can  be  adjusted. 

EARNED  VALUE  AND  SOFTWARE  DEVELOPMENT 
It  has  been  difficult  to  use  earned  value  methods  on  software  develop¬ 
ment  projects.  Models  that  predict  the  amount  and  timing  of  software 
development  costs,  and  metrics  for  accurately  measuring  work  accom¬ 
plishment  have  been  inadequate.  An  obvious  metric,  percentage  of  code 
written,  is  both  deficient  and  misleading.  For  earned  value  methods  to 
be  effective,  BCWS  and  BCWP  must  be  reflect  the  timing  and  technical 
requirements  of  the  work.  Software  development  involves  much  more 
than  writing  code,  and  the  most  difficult  coding  is  often  accomplished 
last.  Therefore,  using  the  percentage  of  code  written  as  an  arbitrary 
method  to  plan  BCWS  and  record  BCWP  would  not  be  an  appropriate 
application  of  the  earned  value  coneept. 

Fortunately,  there  are  more  appropriate  methods  or  metrics  for  plan¬ 
ning  and  measuring  software  development  costs.  Some  of  these  can  be 
used  to  adequately  plan  BCWS,  and  measure  BCWP  and  ACWP.  Re¬ 
gardless  of  the  metric,  the  general  approach  is  to  divide  the  work  into 
portions,  establish  a  schedule  and  a  budget  for  each  portion,  and  then 
use  this  time-phased  budget  as  baseline  against  which  performance  is 
measured. 

Figures  2  and  3  illustrate  how  software  projects  are  planned.  Figure  2 
represents  a  typical  hierarchical  breakdown  of  a  system  into  hardware 
configuration  items  (HWCIs)  and  computer  software  configuration  items 
(CSCls).  CSCIs  arc  divided  into  computer  software  components  (CSCs) 
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Figure  2.  A  System  Hierarchy  for  Software  Development 


and  computer  software  units  (CSUs),  which  represent  the  lowest-level 
subfunctions  of  the  software  (DoD,  1988,  February).  For  performance 
measurement  to  be  meaningful,  performance  and  actual  costs  should  be 
planned  and  measured  where  work  is  being  performed.  For  software 
development  projects,  this  should  be  at  the  CSCI  level  or  below.  At 
higher  levels,  the  planning  of  BCWS  and  the  measurement  of  BCWP 
and  ACWP  would  require  rather  arbitrary  and  subjective  estimates  of 
actual  progress  and  costs. 

To  facilitate  the  objective  measurement  of  progress  and  costs,  earned 
value  methods  typically  require  the  use  of  work  packages.  Figure  3  illus¬ 
trates  the  typical  software  development  process,  known  as  the  “water¬ 
fall”  model  described  in  DOD  Standard  2167A  (DoD,  1988,  February). 
Each  phase  of  this  process  may  be  considered  a  work  package,  appropri¬ 
ate  for  earned  value  application.  The  second  through  seventh  phases  are 
performed  at  the  CSCI  level.  Coding  does  not  begin  until  the  fifth  phase. 
In  the  waterfall  model,  a  coded  product  is  not  available  until  CSCI 
testing  is  completed;  however,  the  completion  of  earlier  phases  is  exten¬ 
sively  documented  and  includes  reviews  and  audits  to  assure  adequacy. 

Using  the  phases  of  software  development  as  work  packages  for  earned 
value  application  appears  to  be  a  viable  approach,  especially  if  the  cost 
and  schedule  of  each  phase  can  be  estimated  with  reasonable  accuracy, 
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SOFTWARE  DEVELOPMENT  PROCESS 

DOD-STD-2167A  PHASES  AND  REVIEWS 
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PHASES 

1 .  System  Requirements  Analysis  and  Design  (RA&D) 

2.  Computer  Software  Configuration  Item  (CSCI)  Requirements 
Analysis  (RA) 

3.  Preliminary  Design 

4.  Detailed  Design 

5.  Code  and  Computer  Software  Unit  (CSU)  Testing 

6.  Computer  Software  Component  (CSC)  Integration  and  Testing 
(l&T) 

7.  Computer  Software  Configuration  Item  (CSCI)  Testing 

8.  System  Testing 

REVIEWS 

System  Design  Review  (SDR) 

Software  Specification  Review  (SSR) 

Preliminary  Design  Review  (PDR) 

Critical  Design  Review  (CDR) 

Test  and  Readiness  Review  (TRR) 

Functional  Configuration  Audit  (FCA) 

Physical  Configuration  Audit  (PCA) 


Figure  3.  The  Software  Development  Process 


and  appropriate  metrics  for  measuring  technical  progress  and  cost  within 
each  phase  are  available.  The  earned  value  method  generally  requires 
that  the  cost  and  schedule  for  each  phase  (work  package)  be  estimated. 
Next,  an  appropriate  metric  to  measure  cost  and  technical  progress  is 
identified  and  used  to  develop  the  time-phased  budget  (BCWS).  Finally, 
as  work  is  accomplished  for  that  work  package,  the  time-phased  budget 
for  the  accomplished  work  is  recorded  as  BCWP  and  its  cost  is  recorded 
as  ACWP. 

Several  models  arc  available  for  predicting  the  cost  and  schedule  for 
each  phase  of  a  software  sy.stem  or  CSCI,  including  the  Constructive 
Cost  Model  (COCOMO).  PRICE-S,  SEER,  SLIM.  SoftCost-R.  and 
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Checkpoint  (Ferens,  1990).  Although  the  accuracies  of  these  models 
have  not  been  validated  for  a  broad  range  of  programs,  they  are  gener¬ 
ally  suitable  for  rough  estimates.  For  a  review  of  the  accuracy  of  these 
models,  see  Institute  of  Electronic  and  Electrical  Engineers  (IEEE) 
(1988). 

Once  the  budget  and  schedule  for  each  work  package  have  been  esti¬ 
mated,  software  metrics  may  be  used  to  plan  BCWS  and  measure  BCWP 
and  ACWP.  Although  much  research  has  been  performed  on  software 
metrics,  there  is  currently  very  little  standardization.  Therefore,  a  man¬ 
ager  must  determine  which  metric  is  appropriate  for  each  phase  of  the 
project. 

There  are  several  desirable  properties  of  software  metrics  (Conte, 
Dunsmorc,  and  Shen,  1986;  DeMarco,  1982;  Humphrey,  1990;  Jones, 
1991).  To  be  useful,  the  metric  should  be  (a)  relevant  to  the  work  being 
measured;  (b)  explicit  (directly  measurable);  (c)  objective;  (d)  absolute 
(able  to  be  assessed  without  reference  to  an  average);  (e)  timely  (avail¬ 
able  early  in  the  project);  and  (f)  independent  from  the  influence  of 
personnel  performing  the  project.  Of  these,  Ayres  and  Rock  (1992) 
found  relevance  to  the  most  important  property.  Accordingly,  the  metrics 
appropriate  for  BCWS,  BCWP  and  ACWP  were  chosen  with  this  prop¬ 
erty  in  mind.  The  first  two  metrics  are  appropriate  for  earned  value 
measurement,  and  the  third  is  most  appropriate  for  ACWP.  The  re¬ 
maining  four  metrics  are  more  useful  in  investigating  variances  than  in 
the  direct  measurement  of  earned  value  or  actual  costs.  Each  metric  and 
its  relevance  to  the  earned  value  approach  is  now  be  briefly  described.  A 
more  detailed  description  of  these  metrics  is  provided  elsewhere  (Ayres 
and  Rock,  1992;  DoD,  1991,  February), 

1.  Requirements  and  Design  Progress.  This  metric  is  based  on  the 
number  of  CSCI  requirements  determined  during  the  first  two 
phases  of  software  development.  The  requirements  are  detailed  in 
several  documents  (System/Segment  Design  Document,  Software 
Requirements  Specification,  Software  Design  Document)  written 
during  these  phases.  As  illustrated  in  Figure  4.  the  planned  and  actual 
CSCI  requirements  are  used  for  determining  BCWS  and  BCWP,  re¬ 
spectively.  Figure  4  also  illustrates  that  the  total  CSCI  requirements 
may  change.  In  addition,  counting  the  requirements  can  be  difficult. 
If  these  limitations  can  be  overcome,  this  metric  is  a  viable  tool  for 
earned  value  application,  especially  early  in  the  project. 

2.  Code  and  Testing  Progress.  This  metric  is  based  on  the  number  of 
eSUs  that  have  been  designed,  coded,  and  tested.  As  illustrated  in 
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Review  SDR  SSR  PDR 


Figure  4.  The  Requirements  and  Design  Process  Metric 


Phase  Detail  Design  Code  and  CSC  CSCI 

CSU  Testing  l&T  Testing 

Review  PDR  CDR  TRR  FCA/PCA 


Figure  5.  The  Code  and  l  est  Progress  Metric 
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Figure  5,  it  is  appropriate  after  the  second  phase  of  the  software 
development  project.  Like  the  previous  metric,  the  planned  and 
actual  CSUs  represent  BCWS  and  BCWP.  In  addition,  the  total 
number  of  planned  CSUs  for  each  phase  represents  the  end  point 
of  the  performance  measurement  baseline  for  that  phase.  Gener¬ 
ally,  this  metric  is  easier  to  measure  than  the  previous  one.  CSU 
progress  can  be  measured  using  a  unit  development  folder  or  simi¬ 
lar  technique.  Also,  more  detailed  information  is  known  about  the 
software  project  in  these  later  phases. 

3.  Person-months  of  Effort.  As  illustrated  in  Figure  6,  this  metric  is 
based  on  person-months  throughout  the  project.  As  such,  it  is 
particularly  useful  for  measuring  ACWP  because  the  costs  of  soft¬ 
ware  development  are  almost  entirely  labor-related.  Using  planned 
person-months  for  BCWS  and  BCWP  is  probably  inappropriate 
because  available  estimation  methods  may  be  inaccurate,  and  the 
time  spent  on  the  project  may  not  correlate  to  progress.  Neverthe¬ 
less.  this  metric  is  useful,  if  only  because  it  is  the  single  metric  in 
this  collection  that  directly  reflects  ACWP. 

4.  Software  Size.  This  metric  tracks  the  size  of  the  software  during 
the  entire  project.  Usually,  size  is  expressed  in  source  lines  of  code 
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Figure  6.  The  Person-Months  Progress  Metric 
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(SLOC).  The  total  size  may  be  divided  into  categories  of  new, 
modified,  and  reused  code.  Since  there  is  a  direct  relationship 
between  size  and  effort  required,  this  metric  may  be  helpful  in 
estimating  actual  cost.  However,  effort  required  and  actual  progress 
may  not  correlate;  accordingly,  the  method  may  be  inadequate  as 
an  earned  value  metric,  and  should  be  used  as  a  technical  param¬ 
eter  to  investigate  the  cause  of  cost  variances  based  on  the  other 
metrics. 

5.  Computer  Resource  Utilization.  This  metric  is  a  measure  of  the 
available  computer  hardware  timing,  memory,  and  input/output  (1/ 
O)  resources  consumed  by  the  software.  It  is  closely  related  to  the 
software  size  metric  described  above  in  that  increases  in  total  size 
will  result  in  a  greater  percentage  of  hardware  resources  utilized. 
Like  software  size,  this  metric  can  be  helpful  early  in  the  program 
for  determining  the  causes  of  variances. 

6.  Requirements  Stability.  This  metric  has  similarities  to  the  require¬ 
ments  and  design  progress  metric.  Like  that  metric,  requirements 
stability  tracks  total  requirements;  however,  it  also  tracks  the  num¬ 
ber  of  changes  (additions,  deletions,  and  modifications)  made  to 
requirements  throughout  the  entire  development  process.  Numer¬ 
ous  or  frequent  changes  will  result  in  additional  effort  required, 
and  may  explain  unfavorable  cost  and  schedule  variances. 

7.  Design  Stability.  This  metric  is  like  requirements  stability  in  that  it 
tracks  the  number  of  changes  to  the  detailed  design  (CSUs).  Like 
the  code  and  testing  progress  metric,  it  is  primarily  useful  later  in 
the  program,  after  preliminary  design  is  completed.  Frequent  lower- 
level  design  changes  will  result  in  additional  effort  required. 

CONCLUSION 

Table  2  lists  the  seven  metrics  described  in  this  paper,  and  indicates  the 
role  that  each  metric  could  have  in  an  earned  value  performance  mea¬ 
surement  system.  The  table  also  indicates  our  judgment  of  how  well  the 
metric  satisfies  the  seven  desirable  properties  of  software  metrics.  Be¬ 
cause  these  properties  are  nearly  identical  to  the  goals  for  earned  value 
measurement  that  are  described  in  C/SCSC,  they  appear  to  be  viable 
candidates  for  earned  value  application,  especially  the  first  three  listed 
in  the  tabic. 

Of  course,  the  seven  metrics  described  in  this  paper  are  not  the  only 
ones.  Especially  worthwhile  arc  “quality  metrics”  that  track  defects,  com- 
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plexity  and  modularity  (Jones,  1991).  While  these  metrics  may  not  di¬ 
rectly  relate  to  earned  value  measurement,  they  do  help  measure  qual¬ 
ity,  which  is  the  sine  qua  non  of  software  projects  today;  using  them  in 
tandem  with  the  ones  recommended  for  earned  value  application  is 
highly  recommended. 


Acquisition  Review  Quarterly 


Spring  1995  -  169 


Using  Earned  Value  for  Performance  Measurement 
on  Software  Development  Projects 


REFERENCES 

Ayres,  Bradley  J.,  and  William  M.  Rock  (1992,  September).  Develop¬ 
ment  of  a  Standard  Set  of  Software  Indicators  for  Aeronautical  Sys¬ 
tems  Center  (AFIT  Thesis  GSS/ENC/92D-1),  Dayton,  Ohio,  Air  Force 
Institute  of  Technology. 

Christensen,  David  S.,  Richard  Antolini,  and  John  W.  McKinney  (1992, 
July).  A  Review  of  Estimate  At  Completion  Research,  Proceedings 
of  the  Society  of  Cost  Estimating  and  Analysis  Society. 

Conte,  S.  D.,  H.  E.  Dunsmore,  and  V.  Y.  Shen  (1986).  Software  Engi¬ 
neering  Metrics  and  Models.  Menlo  Park,  California:  Benjamin- 
Cummings. 

Defense  Systems  Management  College  (1990).  Mission  Critical  Com¬ 
puter  Resources  Management  Guide.  Fort  Belvoir,  Virginia:  Govern¬ 
ment  Printing  Office. 

DeMarco,  Tom  (1982).  Controlling  Software  Projects,  Englewood  Cliffs, 
New  Jersey:  Prentice-Hall. 

Department  of  the  Air  Force  (1987,  October).  Cost! Schedule  Control 
Systems  Criteria  Joint  Implementation  Guide  (Air  Force  Systems  Com¬ 
mand  Pamphlet  173-5).  Washington  DC:  Headquarters  Air  Force 
Systems  Command. 

Department  of  the  Air  Force  (1989,  September).  Guide  to  Analysis  of 
Contractor  Cost  Data  (Air  Force  Systems  Command  Pamphlet  173- 
4).  Washington  DC:  Headquarters  Air  Force  Systems  Command. 

Department  of  Defense  (1991,  February).  Defense  Acquisition  Manage¬ 
ment  Policies  and  Procedures  (Department  of  Defense  Instruction 
5000.2).  Washington  DC. 

Department  of  the  Air  Force  (1991,  November).  Software  Management 
Indicators.  (Air  Force  Pamphlet  800-48).  Washington,  DC. 

Department  of  Defense  (1988,  Februar)').  Defense  System  Software  De¬ 
velopment  (Department  of  Defense  Standard  2167A).  Washington, 
DC. 


170  -  Spring  1995 


Acquisition  Review  Quarterly 


Using  Earned  Value  for  Performance  Measurement 
on  Software  Development  Projects 


Ferens,  Daniel  V.  (1990).  Defense  System  Software  Project  Management. 
Dayton,  Ohio:  Government  Printing  Office. 

Fleming,  Quentin  W.  (1992).  Cost/Schedule  Control  Systems  Criteria  - 
The  Management  Guide  To  CISCSC.  Chicago,  Illinois:  Probus  Pub¬ 
lishing  Company. 

Humphrey,  Watts  S.  (1990).  Managing  the  Software  Process.  Reading, 
Massachuttes:  Addison-Wesley. 

Institute  of  Electronic  and  Electrical  Engineers  (1988,  June).  Standard 
Dictionary  of  Measures  to  Produce  Reliable  Software  (IEEE  Standard 
982.1-1988).  New  York:  Institute  of  Electronic  and  Electrical  Engi¬ 
neers. 

Illinois  Institute  of  Technology  Research  Institute  (1989).  Estimating  the 
Cost  of  Ada  Software  Development,  Lanham,  MD:  IIT  Research  In¬ 
stitute. 

Jones,  Capers  (1991).  Applied  Software  Measurement.  New  York, 
McGraw-Hill. 


Acquisition  Review  Quarterly 


Spring  1995  -  171 


Book  Reviews 


BOOK  REVIEWS 


The 

Balanced 

Budget 


(Malabre.  Jr.,  A.  L.  (1994),  Lost  Prophets  —  An  Insiders  History  of  the  Modem 
Economists.  Boston:  Harvard  Busine.ss  School  Press;  Eisner,  R.  (1994),  The 
Misunderstood  Economy.  Boston:  Harvard  Business  School  Press;  Morton,  D. 
(198.'1)  Making  America  Work  Again.  New  York:  Crown  Publishers;  Brewer,  J. 
(1989),  The  Sinews  of  Power.  New  York:  A.  Kopf;  Krugman,  P.  (1994),  Peddling 
Prosperity,  New  York:  W.W.  Norton  &  Company,  Inc.) 

Reviewed  by 
Franz  A.  P.  Frisch 


^  Y  very  person  listening  to  the  evening  news  or  reading  the  daily  news- 
papers  knows  that  the  words  Balanced  Budget  are  buzz  words  in 
M  ^  Washington.  Congress  is  expected  to  wrestle  with  these  two  words 
for  an  indeterminate  number  of  years  in  an  effort  to  make  them  a  reality. 
The  big  question  is,  of  course,  can  it  work?  Whatever  happens  program 
managers  need  to  he  aware  that  the  discussions  and  ultimate  actions  on 
Capitol  Hill  will  have  an  impact  on  their  programs.  The  reviewer  looks  at 
five  books  that  address  the  Balanced  Budget  concept,  comments  on  the 
authors'  credentials  and  credibility,  and  draws  on  his  own  experience  and 
background  to  present  a  stimulating  view  of  this  thought  consuming  sub¬ 
ject. 

Evers'  economic  theory  is  correct .  .  .  sometimes. 

Every  economic  theory  is  wrong  .  .  .  mostly. 

-Franz  A.  P.  Frisch 


Dr.  Frisch  i.s  a  prolc.ssor  of  .Systems  Acquisition  Management  at  the  De¬ 
fense  Systems  Management  College.  He  holds  Doctor  of  Engineering  de¬ 
gree  from  the  Technical  University  of  Vienna.  Austria. 
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“Book  Reviews”  really  is  not  the  right  heading.  This  is  a  “theme 
review”  of  five  books  written  without  exception  by  economists  of  high 
repute.  All  deal,  at  least  in  one  chapter  of  their  book,  with  the  balanced 
budget  theme,  a  subject  presently  dominating  public  attention  and  de¬ 
bate. 

The  observer  of  economic  events  might  find  highly  irritating  the  way 
in  which  leading  experts  of  the  same  branch  can  disagree,  not  only  in 
details,  but  on  fundamental  principles  and  theories.  But  those  disagree¬ 
ments  illustrate  the  nature  of  economics,  a  discipline  located  somewhere 
between  philosophy  and  science,  demonstrating  the  interplay  between 
abstract  ideas  of  values  and  concrete  measurable  facts.  Hence,  all  eco¬ 
nomic  theories  are  conceptualizations  of  observations  at  a  given  time. 
Unfortunately,  every  conceptualization  and  interpretation  needed  to  ar¬ 
rive  at  a  theory  represents  the  point  of  view  or  more  generally  the  value 
system  of  the  observer  and  objectivity  is  just  an  illusion.  The  same  “facts” 
can  have  different  meanings  for  different  people;  and  even  the  same 
people  may  view  the  facts  differently  depending  on  the  time  and  situa¬ 
tion. 

I  call  this  the  variability  of  opinions  and  judgment.  It  might  seem  like 
a  deviation  from  the  subject  of  the  balanced  budget,  but  I  dare  to  con¬ 
sider  it  a  step  toward  the  core  of  the  problem;  first,  I  remember  the 
stories  I  heard  as  a  child  about  alchemists,  the  people  who  tried  to  make 
gold  for  themselves  and  for  the  kings.  They  tried  it  until  young  science 
at  the  beginning  of  the  age  of  the  enlightenment  proved  it  to  be  impos¬ 
sible.  Later,  when  the  machine  became  a  symbol  of  progress  capturing 
human  imagination,  eager  inventors  searched  for  the  perpetu-mobile, 
the  perpetual  motion  machine,  until  again  progressive  scientists  proved 
that  this  would  be  a  physical  absurdity, 

I  find  interesting  the  story  of  the  alchemists  and  the  inventors  of  the 
perpetual  motion  machine  because  of  a  rather  astounding  fact.  Constant 
failures  to  achieve  those  imagined  goals  have  never  been  accepted  as 
sufficient  proof  that  it  cannot  be  done.  Furthermore,  scientific  proof  has 
been  reluctantly  accepted;  and  without  this  proof,  I  am  convinced  that 
people  would  still  try  to  make  gold  and  to  invent  the  perpetu-mobile. 

And  this  brings  me  back  to  the  balanced  budget,  and  as  a  side  issue,  to 
inflation.  I  remember,  as  a  little  boy  at  home,  listening  in  awe  to  my 
father  and  his  colleagues  discussing  the  budget  and  inflation  at  parties. 
There  was  an  old  professor  of  economic  anthropology  and  economic 
history  who  used  to  say  “Gentlemen,  you  are  barking  up  the  wrong  tree. 
Since  money  exists,  we  have  unbalanced  budgets  and  inflation.  So  why 
not  accept  the  facts?  You  know,  all  secretaries  of  finance,  since  ancient 
times,  tried  both;  they  tried  to  avoid  inflation  and  have  a  balanced  bud- 
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get  and  almost  never  were  they  able  to  do  it.  Does  it  mean  they  were  all 
incompetent  and  stupid?  Or  does  it  mean  both  phenomena  are  embed¬ 
ded  in  (a)  the  nature  of  money  and  (b)  in  the  human  psychology?  If  you 
accept  (a)  and  (b),  then  the  problem  is  not  how  to  solve  the  unsolvable, 
but  to  learn  how  to  live  with  it.” 

I  remember  well  the  long  silence  which  followed  the  old  man’s  remark 
until  a  young  rabbinical  student  started  to  laugh.  “Gentlemen,”  he  said,  “I 
admire  your  questions.  But,  no  offense  intended,  we  Jews  have  known  this 
for  over  2000  years,  when  we  suggested  the  jubilee  of  the  old  testament  to 
cancel  all  debts  eveiy  50  years  and  start  with  a  new  accounting  system.” 

I  cannot  claim  that  I,  as  a  child,  understood  too  much  of  the  conversa¬ 
tions  between  my  father  and  his  friends,  but  I  got  a  feeling  of  uneasi¬ 
ness.  Here  were  a  group  of  highly  educated  men  and  it  seemed  to  me 
that  everyone  had  his  own  opinion,  his  own  point  of  view,  and  defended 
contradicting  positions.  I  must  say,  I  was  sorely  confused.  And  in  school 
sometime  later,  during  the  depression  in  Europe,  I  overheard  the  con¬ 
versation  of  some  teachers  that  made  a  lot  of  sense  to  me.  They  talked 
about  the  budget  and  one  asked  why  the  government  can  have  a  deficit 
when  everybody  else — a  private  person  or  a  company — cannot  spend 
more  than  he  earns.  I  brought  this  wisdom  to  the  attention  of  my  father 
and  had  to  listen  to  his  explanations.  He  started  by  defining  sovereignty, 
an  attribute  to  the  state  in  its  totality,  but  not  to  a  person  or  company, 
living  and  operating  within  the  state.  The  state  (or  the  nation)  can  make 
laws,  have  an  army,  and  foremost  make  money,  but  not  the  entities 
within  the  state. 

Those  memories  from  my  youth  formed  my  thinking  and  mental  pre¬ 
conditioning  toward  all  economic  theories. 

The  first  book  I  selected  for  this  theme  review  was  Malabre’s  Lost 
Prophets-  An  Insider’s  History  of  the  Modem  Economists.  The  reason  I 
started  with  Malabre  is  relatively  simple.  He  is  a  learned  economist  and 
an  economic  reporter  for  the  Wall  Street  Journal.  His  book  is  immensely 
readable,  free  of  professional  jargon,  full  of  humor,  without  preaching 
any  particular  economic  dogma.  He  simply  reports  with  complete  lack 
of  respect,  the  failures  of  the  great  economists  to  predict  the  future.  In 
his  section  “Budgetary  Bafflement”  (page  83)  he  pits  (most  politely) 
experts  against  experts.  He  starts  out  with  the  Eisenhower  administra¬ 
tion  esteem  for  balanced  budgets  and  discusses  the  relationship  between 
the  behavior  of  the  economy  and  the  state  of  the  federal  budget.  He 
says  that  during  the  Kennedy  administration,  this  was  much  more  diffi¬ 
cult  to  evaluate  than  at  the  end  of  the  war  when  the  pent-up  demand  of 
consumers,  flush  with  savings  that  had  accumulated  during  the  war  years, 
was  released. 
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Then,  he  skips  ahead  two  decades  and  refers  to  a  1983  conference, 
sponsored  by  the  Federal  Reserve  Bank  of  Boston  and  organized  to 
address  the  question:  Just  how  much  should  Americans  worry  about  the 
rising  sea  of  red  ink  engulfing  the  federal  budget?  Malabre  calls  this  con¬ 
ference  “unintentionally  hilarious.” 

Reagan  Administration  Treasury  Secretary  Donald  T.  Regan  played 
down  the  importance  of  the  budget  deficit,  but  Martin  Feldstein,  Presi¬ 
dent  Reagan’s  chief  economic  advisor,  warned  that  the  outlook  would 
be  dark  indeed  if  the  red  ink  kept  rising.  Other  speakers  included  Ben¬ 
jamin  M.  Friedman,  a  Harvard  economics  professor  and  a  director  of 
the  National  Bureau  of  Economic  Research,  followed  by  Albert  M. 
Wojnilower,  the  chief  economist  of  First  Boston. 

Friend  Benjamin  (excuse  my  use  of  first  names)  stressed  cause  for 
concern  about  the  unbalanced  budget  and  how  it  would  impede  capital 
formation.  But  Albert,  known  as  a  relative  pessimist  on  economic  pros¬ 
pects  surprised  the  audience  by  stating  that  under  certain  circumstances, 
a  larger  deficit  might  well  be  associated  with  larger  profits  and  invest¬ 
ment.  Albert  concluded  by  saying,  “The  budget  is  like  the  weather:  Ev¬ 
erybody  complains  about  it  but  nobody  does  anything  about  it,  and  no 
one  is  expected  to.”  This  last  remark  supposedly  created  some  friction 
between  Benjamin  and  Albert. 

Malabre  reports  that  another  speaker.  Professor  Robert  Eisner  of 
Northwestern  University,  blamed  the  deficit  essentially  on  inappropri¬ 
ate  accounting  methods  at  the  federal  level  and  argued  that  the  budget 
deficit  was  in  large  measure  an  illusion.  In  particular,  Eisner  explained 
in  his  book.  How  Real  is  the  Federal  Deficit?,  that  a  deficit  that  finances 
construction  of  our  roads,  bridges,  harbors,  and  airports  is  an  invest¬ 
ment  in  the  future.  Expenditures  to  preserve  natural  resources  or  edu¬ 
cate  our  people  and  keep  them  healthy  are  an  investment  in  the  future. 
But.  under  federal  accounting  procedures,  such  investment  is  regarded 
as  additional  red  ink. 

Malabre  reports  more  about  such  differences  of  opinions.  His  section 
about  “Budgetary  Bafflement”  is  both  amusing  and  deeply  disturbing. 
It  .seems  that  the  pro  and  con  expert  groups  are  talking  about  two  en¬ 
tirely  different  subjects: 

•  Eisner,  representing  one  group,  talks  about  the  physical  economy, 
about  bridges  and  airports,  about  construction  and  roads  .  .  .  about 
what  can  be  and  should  be  done. 

•  Feldstein,  representing  the  other  group,  talks  about  the  symbol 
economy,  expressed  in  money. 
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Nothing  demonstrates  the  differences  of  point  of  view  more  drasti¬ 
cally  than  the  Eisner-Feldstein  disagreement,  or  ideological  tunnel  vi¬ 
sion. 

To  me.  the  modern  and  not  so  modem  economy  always  have  two 
sides,  like  the  two  sides  of  a  coin.  The  one  side  is  the  physical  economy 
and  the  other  side  is  the  symbol  economy.  My  colleague  at  DSMC, 
Professor  James  Abellera,  calls  the  layer  in  between  the  ideological  con¬ 
nection  between  the  two.  If  you  accept  my  analogy  with  the  coin,  you 
will  also  accept  the  trivial  fact  that  both  sides  of  the  coin  must  be  the 
same  size.  Think  about  this  for  a  moment  as  a  brain  teaser  and  permit 
me  to  recall  an  event  of  the  history  of  the  Weimer  Republic  between 
1930  and  1933.  Germany  had  more  than  6  million  unemployed.  The 
workers’  unions  requested  an  employment  program  to  be  financed  by 
credits.  The  conservative  government  under  Bruning  refused  in  the  in¬ 
terest  of  a  balanced  budget  and  in  the  election  in  July  1932,  Hitler,  the 
sole  supporter  of  such  a  program,  won. 

This  illustrates  that  the  Eisner-Feldstein  conflict  is  not  necessarily 
new  and  also  illustrates  the  possibility  that  the  right  decision  of  the 
moment  can  be  cata.strophic  in  the  long  run  .  .  .  think  about  it.  Let  me 
close  my  comments  about  Malabre’s  book  with  a  question:  Would  it  not 
be  beautiful  if  we  could  combine  and  coordinate  the  Eisner-Feldstein 
approaches  into  a  single  system  to  everybody’s  benefit? 

Next  I  turned  to  Eisner’s  The  Misunderstood  Economy.  In  particular,  I 
selected  Chapter  5,  titled  “Sense  and  Nonsense  about  Budget  Deficits” 
(page  89). 

The  chapter  starts  with  a  quote  from  John  Maynard  Keynes  book, 
The  General  Theory  of  Employment,  Interest  and  Money.  Then  the  author 
uses  a  1953  quotation  from  President  Eisenhower  relating  the  budget  to 
unemployment  and  the  government’s  responsibility  to  fight  it  as  much  as 
possible.  Next,  he  addresses  balanced  budget  ideas  of  the  Democrats, 
the  Republicans  and  Ross  Perot  and  asks  a  Gallup  Poll  question:  “Which 
is  more  important,  creating  jobs  or  reducing  the  deficit?”  Sixty-five  per¬ 
cent  responded  with  “creating  jobs.” 

Eisner,  at  least  implicitly,  is  talking  at  the  same  time  about  two  re¬ 
lated,  but  different  subjects:  first,  he  talks  about  purely  economical  prob¬ 
lems,  and  second,  about  a  political,  moral  subject.  He  is  most  careful 
with  his  statements  and  always  searches  for  a  balanced  view.  His  discus¬ 
sion  of  measuring  the  deficit,  referring  to  the  difference  between  ac¬ 
counting  principles  in  the  private  and  public  sector  is  most  interesting. 
He  is  saying  that  by  changing  our  accounting  system,  the  deficit  would 
be  not  much  of  a  problem.  If  the  government  were  a  private  company, 
all  past  investments  in  the  infrastructure,  such  as  roads,  ports,  dams. 
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power  stations,  and  so  forth,  would  be  accounted  as  assets.  Of  course, 
this  could  be  done  in  different  ways:  either  as  past  investment  or  by  its 
market  value  or  replacement  cost.  Eisner  does  not  discuss  the  different 
ways  of  accounting,  which  are  subject  to  the  law  of  the  land.  But  regard¬ 
less  of  the  selected  method,  a  private  company  would  be  immensely 
wealthy  and  to  “be  in  the  red”  almost  a  joke,  because  with  these  enor¬ 
mous  assets  you  could  borrow  almost  any  amount  to  correct  or  obliviate 
temporary  cash-flow  problems,  which  is  implied  in  his  table  J,  by  listing 
the  Debt/GDP  ratio  from  1939  to  1993  with  a  quantum  jump  for  World 
War  II  (WWIl).  This,  in  turn,  clearly  implies  that  winning  a  war  is  more 
important  than  a  balanced  budget;  again,  wc  are  back  to  a  political- 
moral  issue.  Just  remember  President  Roosevelt’s  words:  “Do  not  worry 
about  the  deficit,  we  owe  it  to  ourself.”  In  a  footnote  he  gives  what  he 
calls,  an  “explanation  with  elementary  algebra.” 

Then  Eisner  asks  two  questions:  “How  do  deficits  hurt? — or  do  they?” 
He  starts  out  with  the  statement:  “What  is  written  and  said  about  the 
damage  done  by  federal  budget  deficits  is  sheer  nonsense,  no  matter 
how  often  repeated.”  He  talks  about  the  position  of  a  sovereign  govern¬ 
ment  and  about  a  repayment  in  cheaper  dollars  .  .  .  after  inflation.  But 
again,  he  is  extremely  careful  in  choosing  his  words.  He  emphasizes  that 
even  a  sovereign  government  cannot  print  money  without  control:  this 
would  lead  to  hyper-inflation  as  experienced  after  World  War  II  in 
Germany,  Austria,  and  Hungary.  However,  a  little  controlled  inflation 
might  be  a  blessing  for  the  borrower,  albeit  a  curse  for  the  lender. 

This  interpretation  is  somewhat  confirmed  by  Eisner’s  next  subtexts: 
“Spending  our  Children’s  Money”  and  “And  Inflation?” 

He  relates  the  spending  of  our  children’s  money  to  taxes  and  interest 
rates  and  states,  “We  are  told  that  large  deficits  will  cause  inflation.  The 
first  answer  to  this  is  that  we  have  had  some  large  deficits  in  the  last 
decade  and  inflation  has  declined  sharply.”  And  when  he  turns  to  defi¬ 
cits,  he  states.  “In  general,  deficits  can  be  too  small  as  well  as  too  large.” 
In  short,  Eisner  implies  that  the  truth  is  somewhere  in  the  middle — like 
almost  everything  in  life.  He  is  essentially  saying  that  while  a  little  bit  of 
a  deficit  is  good,  too  much  or  none  at  all  is  harmful  for  the  economy  of 
a  nation. 

In  the  next  two  subsections,  “Are  deficits  irrelevant?”  and  “How  defi¬ 
cits  do  matter.”  Eisner  disputes  a  school  of  thought  led  by  Harvard’s 
Robert  Barro,  which  argues  that  deficits  essentially  do  not  matter.  Then 
he  lists  David  Ricardo’s  view  that  government  borrowing  instead  of  taxa¬ 
tion  may  increase  the  people’s  after-tax  income.  Next  he  returns  to  the 
mainstream  argument  that  deficits  do  matter  and  refers  to  the  works  of 
Gottfried  Haberler  of  the  conservative  American  Enterprise  Institute 
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and  to  A.C.  Pigan,  a  “classical”  target  of  Keynes  at  Cambridge.  Eisner 
continues  to  recall  the  expectations  for  a  recession  at  the  end  of  World 
War  II  based  on  the  debt/GNP  ratio  of  well  over  100%  in  1946  and  calls 
the  (thank  God)  erroneous  prediction  as  part  of  the  background  and 
motivation  of  the  work  of  Nobel-laureats  Milton  Friedman  and  Franco 
Modigliani  developing  our  modern  theory  of  the  consumption  function, 
which  he  tries  to  explain  in  plain  English  by  giving  a  hypothetical  ex¬ 
ample. 

Eisner’s  arguments  are  often  on  both  sides  of  the  fence:  but  definitely, 
they  should  serve  as  an  incentive  for  the  student  of  the  economy  to  dig  into 
economic  philosophy  and  history.  In  short,  he  fulfills  his  mission  as  a  teacher. 
He  implies  that  absolute  numbers  (in  dollars)  of  property  are  rather  mean¬ 
ingless  and  only  indexed  numbers  (with  constant  purchasing  power)  count; 
because  otherwise,  inflation  might  distort  the  number  game. 

In  the  next  subsection,  “The  Short  Run:  Impact  on  Consumption, 
Output  and  Employment,”  Eisner  provides  graphical  statistics  about 
changes  in  prices,  employment,  and  real  GDP.  He  brings  in  investment- 
aspects  (beside  others)  and  quotes  Oscar  Lange  (1938)  about  an  “opti¬ 
mal  propensity  to  consume.”  He  tries  to  explain  the  interaction  between 
consumption  and  investment  and  the  “crowding  out”  of  investment  be¬ 
cause  “there  is  no  more  capacity  to  increase  both  consumption  and 
investment.”  He  continues  to  talk  about  the  balance  of  international 
payments  related  to  export  and  imports. 

His  arguments  get  more  and  more  involved  and  it  seems  to  me,  he 
wants  most  correctly  to  say  that  anything  and  everything  in  the  economy 
hangs  together.  We  can  never  consider  one  single  aspect  alone  and 
ultimately,  all  is  driven  by  the  psychological  reaction  of  all  people  to  any 
new  situation,  resulting  in  decisions  to  save  or  to  borrow  based  upon 
hope  or  fear  about  the  future. 

In  the  last  subsection,  “Deficits,  Total  National  Saving,  and  our  Fu¬ 
ture,”  Eisner  represents  himself  more  from  a  philosophical  side.  He 
stresses  the  significance  of  public  investment  in  the  infrastructure  and 
the  intangible  investment  in  education,  training,  research,  and  the  basic 
services  of  public  security:  and  again,  he  tries  to  support  his  judgment 
with  graphical  statistics.  Unfortunately,  his  arguments  get  more  involved 
and  sophisticated  to  the  point  where  the  uninitiated  either  ean  accept 
his  argument  in  awe,  or  else  be  completely  baffled,  perplexed,  or  irri¬ 
tated. 

Closing  out  Eisner,  1  must  say  he  presents  the  subject  in  fascinating 
form,  albeit  not  always  easy  to  understand.  He  highlights  economic  his- 
tor\'  in  its  relationship  to  peace  and  war.  So  1  ask  these  questions;  Will 
the  end  of  the  Cold  War  and  our  success  or  failure  to  capitalize  on  the 
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“peace-dividend”  change  again  our  views  about  the  deficit?  And,  what 
will  happen  if  every  developed  nation  has  a  deficit,  like  all  the  members 
of  the  European  community  according  to  the  agreement  of  Maastrich, 
where  the  members  of  the  European  Currency  established  requirements 
no  one  is  able  to  fulfill.  I  will  return  to  this  at  the  end  of  my  review. 

Eisner  seems  to  be  one  of  the  few  professional  economists  without 
tunnel  vision.  He  is  willing  to  consider  throughout  his  book  all  possible 
points  of  view — at  least  where  there  is  some  logic  involved. 

Next  I  looked  at  Making  America  Work  Again.  I  selected  Davis’s 

book  in  order  to  illustrate  how  opinions — and,  of  course,  priorities — can 
change  in  response  to  political  reality.  Making  America  Work  Again  was 
published  12  years  ago  and  it  represents  thinking  at  the  peak  of  the  cold 
war.  The  book  is  a  call  for  victory,  a  call  to  subjugate  all  considerations 
for  the  fight  and  defeat  of  the  Red  Empire  and  the  communist  danger. 
There  are  no  ifs  and  buts.  All  is  clear  and  rudeness  of  expression  has  its 
purpose. 

In  the  subchapter,  “Capitalistic  Socialism:  Taxes,  Budgets,  and  Defi¬ 
cits,”  Davis  describes  the  superiority  of  the  capitalistic  system  to  control 
the  economy  with  taxes,  thereby  eliminating  the  need  for  revolutionary 
upheaval  and  confiscatory  actions.  In  the  next  subsection,  “The  Balanc¬ 
ing  Act:  The  Greatest  Show  in  Town,”  he  indirectly  praises  frugality, 
only  to  be  suspended  in  times  of  war,  but  stresses  that  war-related  defi¬ 
cits  are  seen  as  essential  but  temporary  extraordinary  expenses  irrel¬ 
evant  to  basic  economic  policy.  He  concedes  that  deficits  gradually  be¬ 
come  immeasurably  seductive,  until  the  notion  of  a  balanced  budget 
begins  to  seem  outdated,  conservative,  and  unnecessarily  regressive  and 
the  popularity  of  the  budget  deficit  was  properly  misused  to  gain  politi¬ 
cal  advantage  by  all  parties.  He  calls  the  Nixon  Administration’s  first 
large  deficit  budget  a  fiction,  because  it  was  called  a  “balanced  full- 
employment”  budget;  a  fiction  leading  to  the  totally  imbalanced  behav¬ 
ior  of  the  political  leaders  and  making  the  projections  of  utopian  statis¬ 
tics  a  matter  of  routine.  After  Nixon,  he  attacks  Presidents  Carter  and 
Reagan  for  predicting  a  surplus  and  ending  up  with  an  increased  deficit 
and  blames  both  for  the  same  utopian  economic  projections. 

Thereafter,  social  transfer  payments  are  attacked  until  he  starts  to 
talk  in  a  subsection,  “Vietnam:  War  is  Peace,”  about  military  spending 
in  the  name  of  economic  stability,  describing  it  as  only  another  case  of 
the  cross-eyed  logic  that  transplants  depression  thinking  into  periods  of 
relative  prosperity.  Then  he  refers  to  the  critics  of  President  Roosevelt’s 
New  Deal,  claiming  that  it  was  the  war,  in  fact,  and  not  the  recovery 
program,  that  brought  us  out  of  the  depression.  And  he  accuses  the 
critics  of  ignoring  the  differences  between  financing  wars  and  economic 
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recoveries.  He  ends  with  the  traditional  wisdom  that  it  is  not  possible  to 
have  guns  and  butter  at  the  same  time. 

In  his  subsection,  “The  Pentagon  Years,”  Davis  states  that  defense 
spending  fires  inflation  by  draining  resources  that  might  be  put  to  better 
use  and  that  “our  economic  theorists  tell  us,  and  with  good  reason,  that 
capitalism  does  not  need  a  war  economy  in  order  to  survive.  Depression 
can  be  averted  through  fiscal  and  monetary  policy,  that  is,  tax  cuts  and 
government  (deficit)  spending;  like  in  building  new  factories,  better  roads 
and  schools  and  similar  valuable  things.” 

Next  he  attacks  the  high  overhead  in  the  defense  industry  and  brings 
up  Grumman’s  apparent  failure  and  inability  to  build  efficiently  or  reli¬ 
ably  the  civilian  flexible  bus  sold  to  cities. 

In  his  last  section,  “Targets  for  Planning.”  Davis  concentrates  on  up¬ 
grading  military  manpower,  the  mandatory  draft,  turning  energy  to 
peacetime  production,  the  essentiality  of  profits  for  motivation — but  not 
a  single  word  about  economic  issues  with  regard  to  planning.  Only  at  the 
end  of  his  book  does  he  return  to  economics,  criticizing  Reagan.  Kennedy, 
Johnson,  and  Nixon  for  deficit  spending. 

He  does  not  forget  Milton  Friedman  for  advoeating  indexation  as 
merely  disguising  an  unwillingness  to  accept  discipline  and  closes  with 
“The  Lorelei  of  the  Lafferites.” 

It  is  difficult  to  comment  on  Davis’s  book.  He  seems  to  try  to  please 
the  ultra-conservatives  and  the  ultra-liberals  at  the  same  time.  Many 
readers  will  reach  for  an  antacid,  but  conservatives  at  different  times 
than  liberals.  Regardless  of  political  leaning,  only  a  fool  would  disagree 
that  winning  a  war  is  more  important  than  anything  else.  On  the  other 
hand,  only  a  fool  may  agree  with  his  extreme  views  on  the  economy;  he 
reaches  the  extreme  on  both  ends  of  the  ideological  scale.  Or  docs  he 
just  try  to  win  readers  from  all  sides  of  the  spectrum?  1  do  not  think  so. 
because  the  text  is  of  overwhelming  arrogance. 

For  Davis,  everybody  seems  to  be  a  fool — only  he  is  right.  And  what 
does  he  mean  by  being  right?  Docs  it  mean  a  balanced  budget  under  all 
conditions  or  to  hell  with  the  balanced  budget  when  it  serves  political 
goals?  For  Davis,  no  middle  ground  exists. 

Brewer’s  The  Sinews  of  Power  is  book  number  four  in  our  review. 
Brewer  is  a  former  professor  of  History  and  Literature  at  Harvard  and 
now  at  UCLA.  The  book  is  a  masterpiece,  as  may  be  expected  of  some¬ 
one  of  his  stature  who  has,  at  the  same  time,  a  deep  understanding  of 
the  interactions  between  national  military  power  and  economic  power. 
The  book — more  than  250  pages  of  text,  supported  by  nearly  700  refer¬ 
ences — is  free  of  any  economic  ideology,  but  amply  supported  by  statis¬ 
tics.  in  the  form  of  tables  and  graphs. 


180  -  Spring  1995 


Acquisition  Review  Quarterly 


Book  Reviews 


It  is  a  fascinating  book  about  the  way  Great  Britain  became  the  domi¬ 
nating  world  power  at  the  time.  It  talks  about  the  East  India  Company. 
It  underlines  the  importance  of  economic  and  social  resources — of  capi¬ 
tal  and  labor,  wealth  and  manpower — to  becoming  a  great  power.  Most 
fascinating  is  the  description  of  “The  radical  increase  in  taxation  and  the 
development  of  public  deficit  finance  (a  national  debt)  on  a  unprec¬ 
edented  scale,  and  the  growth  of  a  sizable  public  administration,  de¬ 
voted  to  organizing  the  fiscal  and  military  activities  of  the  state.” 

In  the  introduction,  the  author  says  that  by  today’s  standards,  mea¬ 
sured  on  the  requirements  of  the  modem  International  Monetary  Fund 
(IMF),  Great  Britain  would  have  been  unable  to  get  a  loan. 

The  relationship  between  military  and  political  power  to  financial 
aspects  is  most  interesting.  It  seems  that  history  teaches  us  that  the 
winner  can  be  never  wrong,  the  loser  never  right.  If  Rome  had  lost 
against  Carthage,  the  entire  world  history  would  look  different.  But  I 
am  supposed  to  talk  about  economics,  not  history. 

Krugman’s  Peddling  Prosperity  is  the  fifth  and  last  book  in  this  review. 
It  is  a  pleasant  book  to  read,  written  with  a  lot  of  humor  and  a  minimum 
of  arrogance.  On  the  fly  page,  Krugman  quotes  from  Keynes,  amplifying 
the  power  of  ideas  of  economists  and  philosophers  both  when  they  are 
right  and  when  they  are  wrong  to  the  practical  men.  In  the  preface,  Krugman 
states  that  “the  subject  of  economy  is  harder  than  physics;  luckily  it  is  not 
quite  as  hard  as  sociology.” 

Why  does  Krugman  say  this?  Quite  obviously,  he  refers  to  the  unend¬ 
ing  choices  possible  for  any  economic  action  from  the  simplest  to  the 
most  complex.  Your  preference  for  a  particular  soap  or  a  specific  car, 
your  judgment  of  the  problem  of  unemployment  or  the  value  of  a  bal¬ 
anced  budget — provided  there  is  a  trade-off — will  depend  on  your  social 
position,  religion,  philosophy,  or  world  view.  And  those  options  are  un¬ 
limited  and  unpredictable.  Now  to  Chapter  6  of  his  book,  “The  Budget 
Deficit.” 

Krugman  is  really  not  saying  anything  that  has  not  already  been  in¬ 
cluded  in  the  other  references.  But,  I  think  he  says  it  better  and  clearer. 
And  foremost,  he  abstains  from  rude  judgment  about  the  actors  in 
economy.  In  short,  he  tries  to  act  like  a  gentleman.  He  says  “The  federal 
government  has  run  a  surplus  in  only  one  year  out  of  the  past  thirty. 
Why  blame  Reagan  for  continuing  the  trend?”  Thereafter,  he  concen¬ 
trates  on  the  deficit  trend  in  terms  of  the  size  of  its  debt  relative  to  the 
size  of  the  tax  base.  Krugman  is  willing  to  accept  a  deficit,  provided  it  is 
not  too  large.  “No  extremes  please”  seems  to  me  a  most  reasonable 
position.  He  tries  desperately,  and  mostly  quite  successfully,  to  avoid 
harsh  critique  on  opposite  points  of  view  between  the  liberals  and  the 
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conservatives.  He  simply  prefers  to  compare  opinions  and  the  shift  or 
change  of  opinions.  He  states  that  “once  upon  a  time,  it  was  liberals  who 
were  soft  on  budget  deficits  . . .  liberals  always  wanted  to  spend  more  on 
social  programs,  and  had  trouble  finding  ways  to  pay  for  them.  On  the 
other  hand,  conservatives  were  tight-fisted  types  who  constantly  warned 
about  the  menace  of  government  borrowing.” 

Thereafter,  he  shifts  to  the  supply-siders  and  “once  come  to  power, 
there  was  an  almost  comic  role  reversal:  liberals  became  the  stem  proph¬ 
ets  of  fiscal  doom,  while  George  Bush  adopted  McFerrin’s  ‘Don’t  worry, 
be  happy’  as  his  unofficial  theme  song.”  Too  bad  I  cannot  quote  the 
entire  chapter  in  this  review.  But  I  strongly  recommend  it  as  appropriate 
reading  material.  It  is  unique  in  its  clarity  and  tolerance. 

In  a  subchapter,  Krugman  introduces  the  term  “hidden  deficit,”  as 
supposedly  springing  from  three  sources:  (1)  the  misregulation  of  finan¬ 
cial  institutions  like  saving  and  loan  associations;  (2)  too  little  invest¬ 
ment  in  infrastructure;  and  (3)  too  little  provisions  (or  thinking  ahead) 
about  the  increase  of  retired  people  to  active  workers. 

I  like  to  abstain  from  any  comment  on  the  misregulation  of  the  finan¬ 
cial  institutions.  But  I  think  you  cannot  have  a  laissez  faire  philosophy 
and  government  control  at  the  same  time.  Such  requirement  would  be  a 
logical  contradiction. 

I  fully  accept  the  second  claim,  the  hidden  deficit  resulting  from  too 
little  investment  in  infrastructure.  I  have  noticed  that  whatever  smart 
engineers  build  needs  maintenance.  And  just  as  with  my  car,  proper 
maintenance  might  be  cheaper  than  to  run  the  car  without  maintenance 
until  it  collapses  and  then  buy  a  new  one.  To  be  more  specific,  the 
maintenance  of  the  infrastructure  and  the  existing  dedicated  investments 
are  the  alpha  and  omega  of  a  healthy  economy.  Without  this  mainte¬ 
nance,  any  modern  economy  will  collapse.  And  we  have  an  example  for 
this:  The  former  USSR.  The  often  plentiful  food  production  was  useless 
and  food  rotted  in  warehouses  because  there  was  no  working  distribu¬ 
tion  system  (roads,  railroads,  etc.),  and  some  of  the  most  modern  facto¬ 
ries  dilapidated  rapidly  to  scrap  as  the  maintenance  problem  was  utterly 
ignored.  You  may  call  this  “ideological  stupidity.” 

As  reviewer.  I  have  some  problems  with  (3),  the  relation  of  workers 
to  retirees.  First,  from  a  purely  economic  point  of  view,  we  need  the 
retired  people  as  customers  for  the  products  of  the  workers  (with  in¬ 
creasing  productivity),  and  second,  from  a  moral  point  of  view,  we  can¬ 
not  exterminate  the  retirees  ...  we  still  love  our  parents.  Beside,  this  is 
not  a  prototypically  American  problem.  The  worker/retiree  ratio  is  much 
worse  in  Sweden,  Germany,  Switzerland,  Austria  and  almost  all  West- 
European  countries,  first,  because  of  the  demographic  age  trend,  and 
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second,  because  of  the  rigorous  retirement  age  limits  (mostly  60  for 
women  and  65  for  men). 

Now  a  few  overall  comments:  But  first,  an  apology  may  be  in  order. 
It  might  be  that  I  misused  this  review  to  sneak  in  some  of  my  personal 
views  on  the  subject.  But,  on  the  other  hand,  this  should  be  the  privilege 
of  an  old  teacher,  who  has  never  taught  the  cookbook  of  the  day,  but 
was  foremost  interested  in  bringing  his  students  to  the  point  of  “think¬ 
ing  for  themselves,”  convinced  that  they  can  do  it,  but  seldom  learned  it 
and  rarely  dared  to  practice  it. 

Now  my  final  comments: 

•  First,  I  am  utterly  surprised  that  none  of  the  five  authors  addresses 
the  question  of  where  to  get  the  money  from  for  an  unbalanced 
budget.  A  sovereign  government  can  print  the  money  (with  all  dan¬ 
gers  involved)  or  it  can  borrow  the  money  from  its  own  population 
or  from  foreigners  (with  all  inconveniences  of  later  repayments).  It 
would  be  interesting  to  hear  the  comments  to  this  point  from  ex¬ 
perts  of  different  orientations. 

•  Second,  from  my  lecture  notes  on  “The  Europeans,”  I  like  to  bring 
the  requirement  as  established  in  Maastrich  to  entitle  a  nation  to 
enter  the  Common-Money-Union  of  Europe.  Just  recently,  three 
other  nations  have  been  accepted  in  the  European  Community  (EC), 
but  not  listed  in  the  table.  They  are  Sweden,  Austria,  and  Finland, 
former  members  of  the  European  free  trade  associates  (EFTA). 
None  of  the  12  listed  nations  of  the  EC  was  able  to  satisfy  the  four 
requirements  for  long  term  interests  (A),  the  rate  of  inflation  (B), 
the  national  debt  (C),  or  the  deficit  (D). 

The  table  shows  that  not  one  of  the  12  members  was  able  to  satisfy  all 
four  requirements  and  only  one  member,  Luxembourg,  was  able  to  sat¬ 
isfy  the  debt  and  deficit  requirements.  And  this  brings  me  back  to  my 
introductory  remarks  to  this  review,  talking  about  the  alchemists  and 
the  inventor  of  the  perpetu-mobile. 

Applicable  to  the  balanced  budget,  we  may  ask  the  impertinent  ques¬ 
tion;  if  all  secretaries  of  finance  are  the  epitome  of  incompetence:  or  the 
most  reasonable  question;  if  the  requirement  for  a  balanced  budget 
might  not  be  a  most  unrealistic  pipe  dream.  But,  the  same  question 
about  the  reasonableness  to  expect  a  balanced  budget  could  be  applied 
to  the  reasonableness  to  expect  an  inflation  free  economy.  If  you  are 
interested  in  this  question,  1  recommend  Don  Paarlberg’s  hook.  An  Analy¬ 
sis  and  Histoiy  of  Inflation  (Praeger  1993).  Paarlberg  is  Professor  Emeri- 
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Table  1, 

•♦REQUIREMENTS  TO  ENTER  THE  COMMON-MONEY-UNION 


A  —  LONG  TERM  INTERESTS  (OVER  ONE  YEAR)  9.2% 

B  —  RATE  OF  INFLATION  2.8% 

C  —  NATIONAL  DEBT  IN  %  OF  GNP  60.0% 

D  —  YEARLY  DEFICIT  IN  %  OF  GNP  3.0% 

“DE-FACTO  SITUATION  (1993)* 


“DE-FACTO  SITUATION  (1993)* 
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tus  of  Agricultural  Economics  at  Purdue  University.  He  served  in  the 
administrations  of  Eisenhower,  Nixon,  and  Ford.  Thereafter,  you  may 
draw  your  own  conclusion,  but  I  expect  you  will  ask  the  same  question 
as  I  did. 

Maybe  the  problem  is  not  how  to  avoid  the  unbalanced  budget  and 
inflation,  but  rather  to  learn  how  to  live  with  it  ...  or  do  we  prefer  the 
mental  state  of  the  Alchemists? 

If  you  are  interested  in  how  experts  can  disagree,  I  recommend  read¬ 
ing  the  essay,  “Competitiveness:  A  Dangerous  Obsession”  (Krugman, 
March/April  1994)  and  comments,  “The  Fight  over  Competitiveness” 
(Prestowitz,  et  al.,  July/August  1994),  both  in  Foreign  Affairs.  They  are 
followed  with  a  reply  from  Kurgmann. 

The  essay  and  the  comments  illustrates  the  diversity  of  points  of  view 
or  what  I  called  at  the  beginning  of  the  review  the  “variability  of  opin¬ 
ions  and  judgment.”  You  also  will  understand  my  quotation  at  the  be¬ 
ginning:  “  Every  economic  theory  is  correct  .  .  .  sometimes.  Every  eco¬ 
nomic  theory  is  wrong  .  .  .  mostly.” 
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Material  proposed  for  the  Acquisition  Review  Quarterly  should  reflect 
scholarly  examination,  disciplined  research,  and  empirically  supported 
experience  in  the  fields  of  defense  systems  management  and  acquisition 
management.  While  defense  acquisition  is  our  primary  focus,  material 
addressing  other  fields  of  management  will  be  considered.  Manuscripts 
supporting  the  Defense  Acquisition  University  commitment  to  improving 
the  acquisition  process  and  the  professionalism  of  the  acquisition  workforce 
are  especially  welcome.  Full-length  articles  should  not  exceed  4,000  words 
with  supporting  text  and  graphics.  Articles  of  600-1,500  words  describing 
new  practices,  programs,  or  techniques  may  also  be  considered. 


MANUSCRIPT  PREPARATION 

Authors  should  study  Chapter  4  oiThe  Publication  Manual  of  the  American 
Psychological  Association,  Fourth  Edition  (1994),  for  specific  guidelines 
regarding  the  preparation  and  proper  appearance  of  manuscript  submis¬ 
sions  (a  checklist  for  reviewing  manuscripts  against  these  guidelines  prior 
to  submission  is  included  as  Appendix  B  of  the  manual).  The  manual  itself 
is  available  from  APA  at  750  First  Street ,  N.E.,  Washington,  D.C.  20002  or 
by  calling  (202)  336-5500.  A  charge  of  $  1 9.95  plus  an  additional  handling  fee 
of  $3.50  is  made  for  each  manual,  and  credit  cards  are  accepted. 

Important:  The  ARQ  is  a  refereed  journal  using  a  masked  (or  “blind”) 
review  process.  Authors  should  take  care  to  limit  personal  or  professional 
identification  to  the  title  page  of  the  manuscript  as  specified  in  Section  4.15 
of  the  APA  Publication  Manual. 


MANUSCRIPT  SUBMISSION 

Each  submission  must  contain: 

1 .  A  cover  letter  providing  the  po.stal  address  and  phone  number  of  each 
author,  as  well  as  any  serviceable  fax  numbers  and/or  e-mail  ad¬ 
dresses.  It  should  al.so  state  that  the  article  is  an  original  product  of  the 
author(s)  not  previously  published  nor  under  concurrent  consider¬ 
ation  by  other  publications.  If  copyrighted  material  of  another  author 
or  publication  is  used  within  the  manuscript,  the  written  permission 
of  the  copyright  holder  must  be  attached  to  the  cover. 
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2.  At  least  one  typewritten  or  electronically  printed  original  version  of 
the  manuscript,  with  one  photocopy.  The  manuscript  should  be 
organized  in  the  following  order:  a)  title  page;  b)  abstract;  c)  manu¬ 
script  text;  d)  references;  e)  content  footnotes;  f)  tables;  g)  figure 
captions;  and  h)  figures.  Only  one  table  or  figure  should  appear  on  a 
single  page. 

3.  With  the  exception  of  typewritten  MS,  a  computer  diskette  (either 
3  1/2"  or  5  1/4")  retaining  the  submission  organized  in  the  order  given 
above.  Microsoft  Word,  Excel,  and  Powerpoint  (or  compatible  appli¬ 
cations  such  as  Corel  Draw  or  Harvard  Graphics)  are  preferred.  Text, 
tables,  and  figures  should  be  segregated  in  separate  files.  Save  each 
chart  or  graphic  as  a  separate  file  (do  not  combine  multiple  images  in 
one  file)  using  the  .eps  format  or,  if  absolutely  necessary,  the  .wmf 
format.  For  guidance  regarding  the  submission  of  ASCII  formatted 
files,  see  Appendix  C  of  the  publication  manual. 

Submissions  should  be  sturdily  packaged  and  mailed  to; 
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Each  author  will  receive  confirmation  of  the  receipt  of  his  or  her  submission 
within  48  hours  of  its  arrival.  Submissions  will  be  logged,  numbered,  and 
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Each  submission  will  be  evaluated  as  to  whether  it  a)  makes  an  original, 
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described  in  the  publication  manual.  Authors  may  be  asked  to  make 
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